Character-level Convolutional Networks for Text Classification

Abstract: This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification. We constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results. Comparisons are offered against traditional models such as bag of words, n-grams and their TFIDF variants, and deep learning models such as word-based ConvNets and recurrent neural networks.

Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

Abstract: We address the rating-inference problem, wherein rather than simply decide whether a review is "thumbs up" or "thumbs down", as in previous sentiment analysis work, one must determine an author's evaluation with respect to a multi-point scale (e.g., one to five "stars"). This task represents an interesting twist on standard multi-class text categorization because there are several different degrees of similarity between class labels; for example, "three stars" is intuitively closer to "four stars" than to "one star". We first evaluate human performance at the task. Then, we apply a meta-algorithm, based on a metric labeling formulation of the problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels. We show that the meta-algorithm can provide significant improvements over both multi-class and regression versions of SVMs when we employ a novel similarity measure appropriate to the problem.

The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

Abstract: We introduce a new test of how well language models capture meaning in children's books. Unlike standard language modelling benchmarks, it distinguishes the task of predicting syntactic function words from that of predicting lower-frequency words, which carry greater semantic content. We compare a range of state-of-the-art models, each with a different way of encoding what has been previously read. We show that models which store explicit representations of long-term contexts outperform state-of-the-art neural language models at predicting semantic content words, although this advantage is not observed for syntactic function words. Interestingly, we find that the amount of text encoded in a single memory representation is highly influential to the performance: there is a sweet-spot, not too big and not too small, between single words and full sentences that allows the most meaningful information in a text to be effectively retained and recalled. Further, the attention over such window-based memories can be trained effectively through self-supervision. We then assess the generality of this principle by applying it to the CNN QA benchmark, which involves identifying named entities in paraphrased summaries of news articles, and achieve state-of-the-art performance.

Dataset statistics.

We use the same tasks and datasets used by the state-of-the-art in speed reading (displayed in Table {{table:9dd9238f-bd04-453e-abe5-390068c7895d}} ), and evaluate against all 5 state-of-the-art models {{cite:74b9fdc}}, {{cite:c581bf5}}, {{cite:f9034bb}}, {{cite:24af983}}, {{cite:989d01a}} in addition to a vanilla LSTM full reading baseline.


TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

Abstract: Vision and language understanding has emerged as a subject undergoing intense study in Artificial Intelligence. Among many tasks in this line of research, visual question answering (VQA) has been one of the most successful ones, where the goal is to learn a model that understands visual content at region-level details and finds their associations with pairs of questions and answers in the natural language form. Despite the rapid progress in the past few years, most existing work in VQA have focused primarily on images. In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways. First, we propose three new tasks designed specifically for video VQA, which require spatio-temporal reasoning from videos to answer questions correctly. Next, we introduce a new large-scale dataset for video VQA named TGIF-QA that extends existing VQA work with our new tasks. Finally, we propose a dual-LSTM based approach with both spatial and temporal attention, and show its effectiveness over conventional VQA techniques through empirical evaluations.

Video Question Answering via Attribute-Augmented Attention Network Learning

Abstract: Video Question Answering is a challenging problem in visual information retrieval, which provides the answer to the referenced video content according to the question. However, the existing visual question answering approaches mainly tackle the problem of static image question, which may be ineffectively for video question answering due to the insufficiency of modeling the temporal dynamics of video contents. In this paper, we study the problem of video question answering by modeling its temporal dynamics with frame-level attention mechanism. We propose the attribute-augmented attention network learning framework that enables the joint frame-level attribute detection and unified video representation learning for video question answering. We then incorporate the multi-step reasoning process for our proposed attention network to further improve the performance. We construct a large-scale video question answering dataset. We conduct the experiments on both multiple-choice and open-ended video question answering tasks to show the effectiveness of the proposed method.

Dataset statistics of four VideoQA benchmark datasets. The columns from left to right indicate dataset name, feature types, vocabulary size, sampled video length, number of videos, size of QA splits, answer set size (Ans size) for open-ended questions, and number of options for multiple-choice questions (MC num).

Question representation.
Each VideoQA dataset has a pre-defined vocabulary which is composed of the top {{formula:0ac9f35b-d60a-42a9-b5e3-250254304aa1}}  most frequent words in the training set. The vocabulary size {{formula:bf380782-7696-4ab2-8ef9-717ea82fede8}}  of each dataset is shown in Table {{table:ba951840-e202-4e8a-85c7-7f759007ff13}} .
We represent each word as a fixed-length learnable word embedding and initialize with the pre-trained GloVe 300-D {{cite:0ddafb9}} feature. We denote the question embedding as a sequence of word embeddings {{formula:e8905180-51cb-4d5d-a199-135d9d214053}} , in which {{formula:2bb1c10c-2e7d-48b3-9aa2-e46b99a7261a}}  is number of words in the question.
We use another LSTM encoder to process question embedding {{formula:8cfb2f05-0fe1-40ef-9161-10c5d6c0312f}} , as highlighted in red in Fig. {{figure:9e33e94c-789d-4327-a819-9c8965f48500}} . The outputs are the encoded text features {{formula:6a109804-f111-4c80-9287-da74250fef3c}}  .

and train the entire network end-to-end.
The intuition of {{formula:63c8c503-0a4a-4dbc-9c36-db3bfa01281c}}  is that the score of the true QA pair should be larger than any negative pair by a margin {{formula:6da724b7-8beb-4304-859f-c393420d91ad}} . During testing, we choose the answer of highest score as the prediction. In Table {{table:ba951840-e202-4e8a-85c7-7f759007ff13}} , we list the number of choices {{formula:2531ca8c-da95-4912-b139-116cd0049e20}}  for each dataset.

In Table {{table:ba951840-e202-4e8a-85c7-7f759007ff13}} , we show the statistics of the four VideoQA benchmark datasets and the experimental settings from their original paper including feature types, vocabulary size, sampled video length, number of videos, size of QA splits, answer set size for open-ended questions, and number of options for multiple-choice questions.


Increases cache hit ratio and delivery delay performance

Hungarian algorithm and successive convex approximation method

Proposed two on-peak hour cache content placement strategies

Propose a novel joint resource allocation and cooperative caching scheme

Proposed long-term caching placement and resource allocation algorithm

Improves cache hit and reduces delivery outage probability

Considerably reduce network cost compared to non-cooperative OMA

Trade-off between performance and computational complexity exist

NOMA Assisted Wireless Caching: Strategies and Performance Analysis

Abstract: Conventional wireless caching assumes that content can be pushed to local caching infrastructure during off-peak hours in an error-free manner; however, this assumption is not applicable if local caches need to be frequently updated via wireless transmission. This paper investigates a new approach to wireless caching for the case when cache content has to be updated during on-peak hours. Two non-orthogonal multiple access (NOMA) assisted caching strategies are developed, namely the push-then-deliver strategy and the push-and-deliver strategy. In the push-then-deliver strategy, the NOMA principle is applied to push more content files to the content servers during a short time interval reserved for content pushing in on-peak hours and to provide more connectivity for content delivery, compared to the conventional orthogonal multiple access (OMA) strategy. The push-and-deliver strategy is motivated by the fact that some users' requests cannot be accommodated locally and the base station has to serve them directly. These events during the content delivery phase are exploited as opportunities for content pushing, which further facilitates the frequent update of the files cached at the content servers. It is also shown that this strategy can be straightforwardly extended to device-to-device caching, and various analytical results are developed to illustrate the superiority of the proposed caching strategies compared to OMA based schemes.

Joint Radio Resource Allocation and Cooperative Caching in PD-NOMA-Based HetNets

Abstract: In this paper, we propose a novel joint resource allocation and cooperative caching scheme for power-domain non-orthogonal multiple access (PD-NOMA)-based heterogeneous networks (HetNets). In our scheme, the requested content is fetched directly from the edge if it is cached in the storage of one of the base stations (BSs), and otherwise is fetched via the backhaul. Our scheme consists of two phases: 1) Caching phase where the contents are saved in the storage of the BSs; and 2) Delivery phase where the requested contents are delivered to users. We formulate a novel optimization problem over radio resources and content placement variables. We aim to minimize the network cost subject to quality-of-service (QoS), caching, subcarrier assignment, and power allocation constraints. By exploiting advanced optimization methods, such as alternative search method (ASM), Hungarian algorithm, successive convex approximation (SCA), we obtain an efficient sub-optimal solution of the optimization problem. Numerical results illustrate that our ergodic caching policy via the proposed resource management algorithm can achieve a considerable reduction on the total cost on average compared to the most popular caching and random caching policy. Moreover, our cooperative NOMA scheme outperforms orthogonal multiple access (OMA) in terms of the delivery cost in general with an acceptable complexity increase.

Caching Placement and Resource Allocation for Cache-Enabling UAV NOMA Networks

Abstract: This article investigates the cache-enabling unmanned aerial vehicle (UAV) cellular networks with massive access capability supported by non-orthogonal multiple access (NOMA). The delivery of a large volume of multimedia contents for ground users is assisted by a mobile UAV base station, which caches some popular contents for wireless backhaul link traffic offloading. In cache-enabling UAV NOMA networks, the caching placement of content caching phase and radio resource allocation of content delivery phase are crucial for network performance. To cope with the dynamic UAV locations and content requests in practical scenarios, we formulate the long-term caching placement and resource allocation optimization problem for content delivery delay minimization as a Markov decision process (MDP). The UAV acts as an agent to take actions for caching placement and resource allocation, which includes the user scheduling of content requests and the power allocation of NOMA users. In order to tackle the MDP, we propose a Q-learning based caching placement and resource allocation algorithm, where the UAV learns and selects action with \emph{soft ${\varepsilon}$-greedy} strategy to search for the optimal match between actions and states. Since the action-state table size of Q-learning grows with the number of states in the dynamic networks, we propose a function approximation based algorithm with combination of stochastic gradient descent and deep neural networks, which is suitable for large-scale networks. Finally, the numerical results show that the proposed algorithms provide considerable performance compared to benchmark algorithms, and obtain a trade-off between network performance and calculation complexity.

Cache placement for cache-aided-NOMA systems

A few cache placement techniques of NOMA systems has been summarised in Table {{table:443f7830-ba35-4576-bc0f-f4d7ba46f5fb}} . In {{cite:b4e4dda}}, the authors proposed a cooperative NOMA- caching scheme to analyze the effect of physical storage of BS and available radio resource parameters like QoS, subcarrier assignment, and power allocation constraints in the network cost. The authors also validated that the proposed scheme can improve the network cost reduction compared to other caching strategies and OMA. An optimization problem for content placement in the NOMA system has been formulated to minimize the average transmit power taking into account cache capacity constraints {{cite:3454ad0}}. In {{cite:452e865}}, the authors have optimized the cache placement strategy to reduce the average delay under the constraint of the storage capacity of a fog-computing AP. Xiang et al. proposed a coded caching delivery strategy and derived optimal transmit power and rate allocation based on cache status, file sizes, and channel conditions to minimize content delivery latency in cellular networks {{cite:3ec4d65}}. A joint cache content popularity prediction and access mode selection problem are formulated as the Stackelberg game in cache-aided NOMA-based F-RANs {{cite:0188807}}. It is observed that a cache-aided network conventionally stores cache contents during an off-peak time, which may not be an efficient approach, particularly when a network frequently needs to update the content of the cache. Ding et al. proposed NOMA as the best candidate for efficiently storing cache contents during on-peak hours and developed two algorithms, push-then-deliver strategy and the push-and-deliver strategy {{cite:c5ec11f}}, {{cite:84d8c73}}. The push-then-deliver strategy stores popular content during on-peak times and delivers when requested. The push-and-deliver strategy deals with the scenario when a user's requested content is not cached.


Generation is an inaccurate synthesis of extraction

pinkneural machine translation is like statistical machine translation in that they are both types of mt systems and have had their share in making mt very popular among the masses.

pinkbeam search is like importance sampling in that they are both methods that consider only the approximate-top-k taggings from an exponential set and require the construction of a good proposal distribution.

pinkvae is like gan in that they are both generative models that have been applied to various scenarios due to their impressive performance and since the arterial and venous phases form different data distribution, to build a relation between them refers to the domain adaptation using generative models.

JUMT at WMT2019 News Translation Task: A Hybrid approach to Machine Translation for Lithuanian to English

Abstract: In the current work, we present a description of the system submitted to WMT 2019 News Translation Shared task. The system was created to translate news text from Lithuanian to English. To accomplish the given task, our system used a Word Embedding based Neural Machine Translation model to post edit the outputs generated by a Statistical Machine Translation model. The current paper documents the architecture of our model, descriptions of the various modules and the results produced using the same. Our system garnered a BLEU score of 17.6.

Neural Particle Smoothing for Sampling from Conditional Sequence Models

Abstract: We introduce neural particle smoothing, a sequential Monte Carlo method for sampling annotations of an input string from a given probability model. In contrast to conventional particle filtering algorithms, we train a proposal distribution that looks ahead to the end of the input string by means of a right-to-left LSTM. We demonstrate that this innovation can improve the quality of the sample. To motivate our formal choices, we explain how neural transduction models and our sampler can be viewed as low-dimensional but nonlinear approximations to working with HMMs over very large state spaces.

Phase Collaborative Network for Two-Phase Medical Image Segmentation

Abstract: In real-world practice, medical images acquired in different phases possess complementary information, {\em e.g.}, radiologists often refer to both arterial and venous scans in order to make the diagnosis. However, in medical image analysis, fusing prediction from two phases is often difficult, because (i) there is a domain gap between two phases, and (ii) the semantic labels are not pixel-wise corresponded even for images scanned from the same patient. This paper studies organ segmentation in two-phase CT scans. We propose Phase Collaborative Network (PCN), an end-to-end framework that contains both generative and discriminative modules. PCN can be mathematically explained to formulate phase-to-phase and data-to-label relations jointly. Experiments are performed on a two-phase CT dataset, on which PCN outperforms the baselines working with one-phase data by a large margin, and we empirically verify that the gain comes from inter-phase collaboration. Besides, PCN transfers well to two public single-phase datasets, demonstrating its potential applications.

Error analysis of ACCoRD system output: generation stage. Common generation error types for a sample ({{formula:5a32fdbb-be6d-444f-819e-92a572101b7b}} ) of ACCoRD system outputs include generations that are an inaccurate synthesis of source text; generations that are technically correct but unhelpful because they lack important details; incoherent generations; and instances where the second sentence of extraction is appended verbatim, often resulting in a difficult to read, incoherent, or inaccurate generation. A final error type not shown was, appending irrelevant information, which occurred in 5% of the sample. Multiple error types can be present in the same generation.

An analysis of a sample ({{formula:3d35cf9d-6746-45e3-9605-ca44f282e6e1}} ) of ACCoRD system outputs revealed particular areas for improvement. {{formula:7ee2e189-f4e4-405f-a015-a2ea7afa3cb0}}  of the sample contained at least type of error.
The majority of the error came from the generation component of our system and included generations that were an inaccurate synthesis of extracted context; generations that were technically correct, but unhelpful because they lacked important details; incoherent generations; instances where the second sentence of extraction was appended verbatim, often resulting in one of the other error types; and generations where irrelevant details were appended (see Table {{table:99d0a1f5-9270-4aa3-9a1c-211cb987ad48}}  for proportions and an example of each error type).
In a minority of cases, system errors were due to issues at the extractive stage. These error types included poor delimitation of the target ForeCite concept within candidate extractions that resulted in an inaccurate generation; low quality extracted text that made it difficult to produce a high quality generation; and difficulties in delimiting sentences in scientific text using available methods (see Table {{table:93f29535-2a6f-47f9-accd-2019d5681a18}} ). While these errors are of varying degrees of severity, further work on methods that address these issues at scale will be important to expanding the proof-of-concept system we propose here.


Achieving a secure and practical decentralized data storage solution in edge computing.

Secure and efficient data trading environment in IoT.

Proposing a solution framework for anonymity, integrity and adaptability of data storage by reviewing the challenges and requirements of deployment of blockchain and edge computing in IoT systems and implementing a system prototype utilizing Ethereum's JavaScript API.

Presenting a data trading platform using Ethereum with consensus of proof-of-concept (PoC).

Blockchain-Based Secure Storage Management with Edge Computing for IoT

Abstract: As a core technology to manage decentralized systems, blockchain is gaining much popularity to deploy such applications as smart grid and healthcare systems. However, its utilization in resource-constrained mobile devices is limited due to high demands of resources and poor scalability with frequent-intensive transactions. Edge computing can be integrated to facilitate mobile devices in offloading their mining tasks to cloud resources. This integration ensures reliable access, distributed computation and untampered storage for scalable and secure transactions. It is imperative therefore that crucial issues of security, scalability and resources management be addressed to achieve successful integration. Studies have been conducted to explore suitable architectural requirements, and some researchers have applied the integration to deploy some specific applications. Despite these efforts, however, issues of anonymity, adaptability and integrity still need to be investigated further to attain a practical, secure decentralized data storage. We based our study on peer-to-peer and blockchain to achieve an Internet of Things (IoT) design supported by edge computing to acquire security and scalability levels needed for the integration. We investigated existing blockchain and associated technologies to discover solutions that address anonymity, integrity and adaptability issues for successful integration of blockchain in IoT systems. The discovered solutions were then incorporated in our conceptual design of the decentralized application prototype presented for secure storage of IoT data and transactions.

Edge Computing to Secure IoT Data Ownership and Trade with the Ethereum Blockchain

Abstract: With an increasing penetration of ubiquitous connectivity, the amount of data describing the actions of end-users has been increasing dramatically, both within the domain of the Internet of Things (IoT) and other smart devices. This has led to more awareness of users in terms of protecting personal data. Within the IoT, there is a growing number of peer-to-peer (P2P) transactions, increasing the exposure to security vulnerabilities, and the risk of cyberattacks. Blockchain technology has been explored as middleware in P2P transactions, but existing solutions have mainly focused on providing a safe environment for data trade without considering potential changes in interaction topologies. we present EdgeBoT, a proof-of-concept smart contracts based platform for the IoT built on top of the ethereum blockchain. With the Blockchain of Things (BoT) at the edge of the network, EdgeBoT enables a wider variety of interaction topologies between nodes in the network and external services while guaranteeing ownership of data and end users’ privacy. in EdgeBoT, edge devices trade their data directly with third parties and without the need of intermediaries. This opens the door to new interaction modalities, in which data producers at the edge grant access to batches of their data to different third parties. Leveraging the immutability properties of blockchains, together with the distributed nature of smart contracts, data owners can audit and are aware of all transactions that have occurred with their data. we report initial results demonstrating the potential of EdgeBoT within the IoT. we show that integrating our solutions on top of existing IoT systems has a relatively small footprint in terms of computational resource usage, but a significant impact on the protection of data ownership and management of data trade.

Data Management Framework for IBEC

The above-mentioned studies on data management framework for the IBEC are summarized in Table {{table:ecf7591d-e872-4290-a3b6-387c9b635e65}} .


Represents code by a collection of paths between terminal nodes n the AST

Generates code's representation by AST path

Predicts the methods and API calls by flattened AST

Generates code through LSTM or pointer network

Feeds different paths to Transformer-XL with multi-task learning

Feeds different paths with up/down momentum to Word2vec

A generated model on ExprGen task which first obtains a graph by attribute grammars and later compute the attribute representations with GGNN and GRU

Uses AST Graph Attention Block(ASTGab) to model the flattened sequence of AST to capture different dependencies

Constructs graphs by adding different types of edges and use GGNN to learn the representation of the graph

Introduces a Graph-Structured Cache representing vocabulary words as additional nodes, uses MPNN and CharCNN to generate outputs to further address open vocabulary issue

code2seq: Generating Sequences from Structured Representations of Code

Abstract: The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval. Sequence-to-sequence (seq2seq) models, adopted from neural machine translation (NMT), have achieved state-of-the-art performance on these tasks by treating source code as a sequence of tokens. We present ${\rm {\scriptsize CODE2SEQ}}$: an alternative approach that leverages the syntactic structure of programming languages to better encode source code. Our model represents a code snippet as the set of compositional paths in its abstract syntax tree (AST) and uses attention to select the relevant paths while decoding. We demonstrate the effectiveness of our approach for two tasks, two programming languages, and four datasets of up to $16$M examples. Our model significantly outperforms previous models that were specifically designed for programming languages, as well as state-of-the-art NMT models. An interactive online demo of our model is available at http://code2seq.org. Our code, data and trained models are available at http://github.com/tech-srl/code2seq.

code2vec: learning distributed representations of code

Abstract: We present a neural model for representing snippets of code as continuous distributed vectors (``code embeddings''). The main idea is to represent a code snippet as a single fixed-length code vector, which can be used to predict semantic properties of the snippet. To this end, code is first decomposed to a collection of paths in its abstract syntax tree. Then, the network learns the atomic representation of each path while simultaneously learning how to aggregate a set of them. We demonstrate the effectiveness of our approach by using it to predict a method's name from the vector representation of its body. We evaluate our approach by training a model on a dataset of 12M methods. We show that code vectors trained on this dataset can predict method names from files that were unobserved during training. Furthermore, we show that our model learns useful method name vectors that capture semantic similarities, combinations, and analogies. A comparison of our approach to previous techniques over the same dataset shows an improvement of more than 75%, making it the first to successfully predict method names based on a large, cross-project corpus. Our trained model, visualizations and vector similarities are available as an interactive online demo at http://code2vec.org. The code, data and trained models are available at https://github.com/tech-srl/code2vec.

Pythia: AI-assisted Code Completion System

Abstract: In this paper, we propose a novel end-to-end approach for AI-assisted code completion called Pythia. It generates ranked lists of method and API recommendations which can be used by software developers at edit time. The system is currently deployed as part of Intellicode extension in Visual Studio Code IDE. Pythia exploits state-of-the-art large-scale deep learning models trained on code contexts extracted from abstract syntax trees. It is designed to work at a high throughput predicting the best matching code completions on the order of 100 ms. We describe the architecture of the system, perform comparisons to frequency-based approach and invocation-based Markov Chain language model, and discuss challenges serving Pythia models on lightweight client devices. The offline evaluation results obtained on 2700 Python open source software GitHub repositories show a top-5 accuracy of 92%, surpassing the baseline models by 20% averaged over classes, for both intra and cross-project settings.

Code Completion with Neural Attention and Pointer Networks

Abstract: Intelligent code completion has become an essential research task to accelerate modern software development. To facilitate effective code completion for dynamically-typed programming languages, we apply neural language models by learning from large codebases, and develop a tailored attention mechanism for code completion. However, standard neural language models even with attention mechanism cannot correctly predict the out-of-vocabulary (OoV) words that restrict the code completion performance. In this paper, inspired by the prevalence of locally repeated terms in program source code, and the recently proposed pointer copy mechanism, we propose a pointer mixture network for better predicting OoV words in code completion. Based on the context, the pointer mixture network learns to either generate a within-vocabulary word through an RNN component, or regenerate an OoV word from local context through a pointer component. Experiments on two benchmarked datasets demonstrate the effectiveness of our attention mechanism and pointer mixture network on the code completion task.

Code Prediction by Feeding Trees to Transformers

Abstract: Code prediction, more specifically autocomplete, has become an essential feature in modern IDEs. Autocomplete is more effective when the desired next token is at (or close to) the top of the list of potential completions offered by the IDE at cursor position. This is where the strength of the underlying machine learning system that produces a ranked order of potential completions comes into play. We advance the state-of-the-art in the accuracy of code prediction (next token prediction) used in autocomplete systems. Our work uses Transformers as the base neural architecture. We show that by making the Transformer architecture aware of the syntactic structure of code, we increase the margin by which a Transformer-based system outperforms previous systems. With this, it outperforms the accuracy of several state-of-the-art next token prediction systems by margins ranging from 14% to 18%. We present in the paper several ways of communicating the code structure to the Transformer, which is fundamentally built for processing sequence data. We provide a comprehensive experimental evaluation of our proposal, along with alternative design choices, on a standard Python dataset, as well as on Facebook internal Python corpus. Our code and data preparation pipeline will be available in open source.

A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning

Abstract: Code completion, one of the most useful features in the Integrated Development Environments (IDEs), can accelerate software development by suggesting the libraries, APIs, and method names in real-time. Recent studies have shown that statistical language models can improve the performance of code completion tools through learning from large-scale software repositories. However, these models suffer from three major drawbacks: a) The hierarchical structural information of the programs is not fully utilized in the program's representation; b) In programs, the semantic relationships can be very long. Existing recurrent neural networks based language models are not sufficient to model the long-term dependency. c) Existing approaches perform a specific task in one model, which leads to the underuse of the information from related tasks. To address these challenges, in this paper, we propose a self-attentional neural architecture for code completion with multi-task learning. To utilize the hierarchical structural information of the programs, we present a novel method that considers the path from the predicting node to the root node. To capture the long-term dependency in the input programs, we adopt a self-attentional architecture based network as the base language model. To enable the knowledge sharing between related tasks, we creatively propose a Multi-Task Learning (MTL) framework to learn two related tasks in code completion jointly. Experiments on three real-world datasets demonstrate the effectiveness of our model when compared with state-of-the-art methods.

A General Path-Based Representation for Predicting Program Properties

Abstract: Predicting program properties such as names or expression types has a wide range of applications. It can ease the task of programming and increase programmer productivity. A major challenge when learning from programs is $\textit{how to represent programs in a way that facilitates effective learning}$. We present a $\textit{general path-based representation}$ for learning from programs. Our representation is purely syntactic and extracted automatically. The main idea is to represent a program using paths in its abstract syntax tree (AST). This allows a learning model to leverage the structured nature of code rather than treating it as a flat sequence of tokens. We show that this representation is general and can: (i) cover different prediction tasks, (ii) drive different learning algorithms (for both generative and discriminative models), and (iii) work across different programming languages. We evaluate our approach on the tasks of predicting variable names, method names, and full types. We use our representation to drive both CRF-based and word2vec-based learning, for programs of four languages: JavaScript, Java, Python and C\#. Our evaluation shows that our approach obtains better results than task-specific handcrafted representations across different tasks and programming languages.

Generative Code Modeling with Graphs

Abstract: Generative models for source code are an interesting structured prediction problem, requiring to reason about both hard syntactic and semantic constraints as well as about natural, likely programs. We present a novel model for this problem that uses a graph to represent the intermediate state of the generated output. The generative procedure interleaves grammar-driven expansion steps with graph augmentation and neural message passing steps. An experimental evaluation shows that our new model can generate semantically meaningful expressions, outperforming a range of strong baselines.

Code Completion by Modeling Flattened Abstract Syntax Trees as Graphs

Abstract: Code completion has become an essential component of integrated development environments. Contemporary code completion methods rely on the abstract syntax tree (AST) to generate syntactically correct code. However, they cannot fully capture the sequential and repetitive patterns of writing code and the structural information of the AST. To alleviate these problems, we propose a new code completion approach named CCAG, which models the flattened sequence of a partial AST as an AST graph. CCAG uses our proposed AST Graph Attention Block to capture different dependencies in the AST graph for representation learning in code completion. The sub-tasks of code completion are optimized via multi-task learning in CCAG, and the task balance is automatically achieved using uncertainty without the need to tune task weights. The experimental results show that CCAG has superior performance than state-of-the-art approaches and it is able to provide intelligent code completion.

Learning to Represent Programs with Graphs

Abstract: Learning tasks on source code (i.e., formal languages) have been considered recently, but most work has tried to transfer natural language methods and does not capitalize on the unique opportunities offered by code's known syntax. For example, long-range dependencies induced by using the same variable or function in distant locations are often not considered. We propose to use graphs to represent both the syntactic and semantic structure of code and use graph-based deep learning methods to learn to reason over program structures. In this work, we present how to construct graphs from source code and how to scale Gated Graph Neural Networks training to such large graphs. We evaluate our method on two tasks: VarNaming, in which a network attempts to predict the name of a variable given its usage, and VarMisuse, in which the network learns to reason about selecting the correct variable that should be used at a given program location. Our comparison to methods that use less structured program representations shows the advantages of modeling known structure, and suggests that our models learn to infer meaningful names and to solve the VarMisuse task in many cases. Additionally, our testing showed that VarMisuse identifies a number of bugs in mature open-source projects.

Open Vocabulary Learning on Source Code with a Graph-Structured Cache

Abstract: Machine learning models that take computer program source code as input typically use Natural Language Processing (NLP) techniques. However, a major challenge is that code is written using an open, rapidly changing vocabulary due to, e.g., the coinage of new variable and method names. Reasoning over such a vocabulary is not something for which most NLP methods are designed. We introduce a Graph-Structured Cache to address this problem; this cache contains a node for each new word the model encounters with edges connecting each word to its occurrences in the code. We find that combining this graph-structured cache strategy with recent Graph-Neural-Network-based models for supervised learning on code improves the models' performance on a code completion task and a variable naming task --- with over $100\%$ relative improvement on the latter --- at the cost of a moderate increase in computation time.

Code Generation

Code Generation can provide varying levels of granularity of code output depending on the input. Code prediction in IDE that anticipates blank areas of input code snippets, such as method name prediction, next token prediction, and so on, is a typical code generation task. It's also a kind of code generation task to generate snippets of code from natural language. Code generation considerably enhances developers' efficiency and has been extensively researched in both industry and academia. The relevant approaches are summarized in Table {{table:f07ec7c6-ec7d-4d32-a31c-8756db915184}} , which is based mostly on the structure used by the model in the code generation task.


Hawaii International Conference on System Sciences 2022 (HICSS-55)

A Study on Fairness and Trust Perceptions in Automated Decision Making

Abstract: Automated decision systems are increasingly used for consequential decision making -- for a variety of reasons. These systems often rely on sophisticated yet opaque models, which do not (or hardly) allow for understanding how or why a given decision was arrived at. This is not only problematic from a legal perspective, but non-transparent systems are also prone to yield undesirable (e.g., unfair) outcomes because their sanity is difficult to assess and calibrate in the first place. In this work, we conduct a study to evaluate different attempts of explaining such systems with respect to their effect on people's perceptions of fairness and trustworthiness towards the underlying mechanisms. A pilot study revealed surprising qualitative insights as well as preliminary significant effects, which will have to be verified, extended and thoroughly discussed in the larger main study.

Perceptions of Fairness and Trustworthiness Based on Explanations in Human vs. Automated Decision-Making

Abstract: Automated decision systems (ADS) have become ubiquitous in many high-stakes domains. Those systems typically involve sophisticated yet opaque artificial intelligence (AI) techniques that seldom allow for full comprehension of their inner workings, particularly for affected individuals. As a result, ADS are prone to deficient oversight and calibration, which can lead to undesirable (e.g., unfair) outcomes. In this work, we conduct an online study with 200 participants to examine people's perceptions of fairness and trustworthiness towards ADS in comparison to a scenario where a human instead of an ADS makes a high-stakes decision -- and we provide thorough identical explanations regarding decisions in both cases. Surprisingly, we find that people perceive ADS as fairer than human decision-makers. Our analyses also suggest that people's AI literacy affects their perceptions, indicating that people with higher AI literacy favor ADS more strongly over human decision-makers, whereas low-AI-literacy people exhibit no significant differences in their perceptions.

"There Is Not Enough Information": On the Effects of Explanations on Perceptions of Informational Fairness and Trustworthiness in Automated Decision-Making

Abstract: Automated decision systems (ADS) are increasingly used for consequential decision-making. These systems often rely on sophisticated yet opaque machine learning models, which do not allow for understanding how a given decision was arrived at. In this work, we conduct a human subject study to assess people's perceptions of informational fairness (i.e., whether people think they are given adequate information on and explanation of the process and its outcomes) and trustworthiness of an underlying ADS when provided with varying types of information about the system. More specifically, we instantiate an ADS in the area of automated loan approval and generate different explanations that are commonly used in the literature. We randomize the amount of information that study participants get to see by providing certain groups of people with the same explanations as others plus additional explanations. From our quantitative analyses, we observe that different amounts of information as well as people's (self-assessed) AI literacy significantly influence the perceived informational fairness, which, in turn, positively relates to perceived trustworthiness of the ADS. A comprehensive analysis of qualitative feedback sheds light on people's desiderata for explanations, among which are (i) consistency (both with people's expectations and across different explanations), (ii) disclosure of monotonic relationships between features and outcome, and (iii) actionability of recommendations.

Appropriate Fairness Perceptions? On the Effectiveness of Explanations in Enabling People to Assess the Fairness of Automated Decision Systems

Abstract: It is often argued that one goal of explaining automated decision systems (ADS) is to facilitate positive perceptions (e.g., fairness or trustworthiness) of users towards such systems. This viewpoint, however, makes the implicit assumption that a given ADS is fair and trustworthy, to begin with. If the ADS issues unfair outcomes, then one might expect that explanations regarding the system’s workings will reveal its shortcomings and, hence, lead to a decrease in fairness perceptions. Consequently, we suggest that it is more meaningful to evaluate explanations against their effectiveness in enabling people to appropriately assess the quality (e.g., fairness) of an associated ADS. We argue that for an effective explanation, perceptions of fairness should increase if and only if the underlying ADS is fair. In this in-progress work, we introduce the desideratum of appropriate fairness perceptions, propose a novel study design for evaluating it, and outline next steps towards a comprehensive experiment.

A Ranking Approach to Fair Classification

Abstract: Algorithmic decision systems are increasingly used in areas such as hiring, school admission, or loan approval. Typically, these systems rely on labeled data for training a classification model. However, in many scenarios, ground-truth labels are unavailable, and instead we have only access to imperfect labels as the result of (potentially biased) human-made decisions. Despite being imperfect, historical decisions often contain some useful information on the unobserved true labels. In this paper, we focus on scenarios where only imperfect labels are available and propose a new fair ranking-based decision system based on monotonic relationships between legitimate features and the outcome. Our approach is both intuitive and easy to implement, and thus particularly suitable for adoption in real-world settings. More in detail, we introduce a distance-based decision criterion, which incorporates useful information from historical decisions and accounts for unwanted correlation between protected and legitimate features. Through extensive experiments on synthetic and real-world data, we show that our method is fair in the sense that a) it assigns the desirable outcome to the most qualified individuals, and b) it removes the effect of stereotypes in decision-making, thereby outperforming traditional classification algorithms. Additionally, we are able to show theoretically that our method is consistent with a prominent concept of individual fairness which states that “similar individuals should be treated similarly.”

Overview of my dissertation-related academic work to date.

In this section, I briefly summarize main results and contributions of my dissertation-related academic work to date. An overview of peer-reviewed accepted and in-submission work is given in Table {{table:62ecd56f-93c7-4697-9346-73d7eace9acc}} .


Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Abstract: Deep learning techniques for point cloud data have demonstrated great potentials in solving classical problems in 3D computer vision such as 3D object classification and segmentation. Several recent 3D object classification methods have reported state-of-the-art performance on CAD model datasets such as ModelNet40 with high accuracy (~92\%). Despite such impressive results, in this paper, we argue that object classification is still a challenging task when objects are framed with real-world settings. To prove this, we introduce ScanObjectNN, a new real-world point cloud object dataset based on scanned indoor scene data. From our comprehensive benchmark, we show that our dataset poses great challenges to existing point cloud classification techniques as objects from real-world scans are often cluttered with background and/or are partial due to occlusions. We identify three key open problems for point cloud object classification, and propose new point cloud classification neural networks that achieve state-of-the-art performance on classifying objects with cluttered background. Our dataset and code are publicly available in our project page https://hkust-vgd.github.io/scanobjectnn/.

PointDAN: A Multi-Scale 3D Domain Adaption Network for Point Cloud Representation

Abstract: Domain Adaptation (DA) approaches achieved significant improvements in a wide range of machine learning and computer vision tasks (i.e., classification, detection, and segmentation). However, as far as we are aware, there are few methods yet to achieve domain adaptation directly on 3D point cloud data. The unique challenge of point cloud data lies in its abundant spatial geometric information, and the semantics of the whole object is contributed by including regional geometric structures. Specifically, most general-purpose DA methods that struggle for global feature alignment and ignore local geometric information are not suitable for 3D domain alignment. In this paper, we propose a novel 3D Domain Adaptation Network for point cloud data (PointDAN). PointDAN jointly aligns the global and local features in multi-level. For local alignment, we propose Self-Adaptive (SA) node module with an adjusted receptive field to model the discriminative local structures for aligning domains. To represent hierarchically scaled features, node-attention module is further introduced to weight the relationship of SA nodes across objects and domains. For global alignment, an adversarial-training strategy is employed to learn and align global features across domains. Since there is no common evaluation benchmark for 3D point cloud DA scenario, we build a general benchmark (i.e., PointDA-10) extracted from three popular 3D object/scene datasets (i.e., ModelNet, ShapeNet and ScanNet) for cross-domain 3D objects classification fashion. Extensive experiments on PointDA-10 illustrate the superiority of our model over the state-of-the-art general-purpose DA methods.

ShapeNet: An Information-Rich 3D Model Repository

Abstract: We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects. ShapeNet contains 3D models from a multitude of semantic categories and organizes them under the WordNet taxonomy. It is a collection of datasets providing many semantic annotations for each 3D model such as consistent rigid alignments, parts and bilateral symmetry planes, physical sizes, keywords, as well as other planned annotations. Annotations are made available through a public web-based interface to enable data visualization of object attributes, promote data-driven geometric analysis, and provide a large-scale quantitative benchmark for research in computer graphics and vision. At the time of this technical report, ShapeNet has indexed more than 3,000,000 models, 220,000 models out of which are classified into 3,135 categories (WordNet synsets). In this report we describe the ShapeNet effort as a whole, provide details for all currently available datasets, and summarize future plans.

3D object datasets for classification task.

We first pretrain the models by the multi-task learning on Primitive3D. The output feature representations are then evaluated on multiple object classification benchmarks and compared to the models pretrained on other datasets.
We consider four commonly used 3D object datasets for our experiments, namely, ModelNet40 {{cite:bd2ff7c}}, ScanObjectNN {{cite:bc925be}}, ScanNet10 {{cite:10cf834}} and ShapeNet{{cite:de91e0c}}, with the statistics summarized in Table {{table:bc0a1712-a97c-4711-85b7-d8bfed63734a}} . The first three datasets are utilized for both benchmarking and pretraining, while the ShapeNet is included only to serve the pretraining purpose. It is worth mentioning that, compared to the synthesis dataset ModelNet40, the classifications on ScanObjectNN and ScanNet are more challenging due to the presence of real-world noise such as the occlusion of objects. In our comparison, we use the same unsupervised pretraining task on the compared datasets, while different supervised pretraining tasks based on their own annotations. Finally, identical settings are applied to all pretraining tasks.


Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades

Abstract: Creating datasets for Neuromorphic Vision is a challenging task. A lack of available recordings from Neuromorphic Vision sensors means that data must typically be recorded specifically for dataset creation rather than collecting and labelling existing data. The task is further complicated by a desire to simultaneously provide traditional frame-based recordings to allow for direct comparison with traditional Computer Vision algorithms. Here we propose a method for converting existing Computer Vision static image datasets into Neuromorphic Vision datasets using an actuated pan-tilt camera platform. Moving the sensor rather than the scene or image is a more biologically realistic approach to sensing and eliminates timing artifacts introduced by monitor updates when simulating motion on a computer monitor. We present conversion of two popular image datasets (MNIST and Caltech101) which have played important roles in the development of Computer Vision, and we provide performance metrics on these datasets using spike-based recognition algorithms. This work contributes datasets for future use in the field, as well as results from spike-based algorithms against which future works can compare. Furthermore, by converting datasets already popular in Computer Vision, we enable more direct comparison with frame-based approaches.

HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification

Abstract: Event-based cameras have recently drawn the attention of the Computer Vision community thanks to their advantages in terms of high temporal resolution, low power consumption and high dynamic range, compared to traditional frame-based cameras. These properties make event-based cameras an ideal choice for autonomous vehicles, robot navigation or UAV vision, among others. However, the accuracy of event-based object classification algorithms, which is of crucial importance for any reliable system working in real-world conditions, is still far behind their frame-based counterparts. Two main reasons for this performance gap are: 1. The lack of effective low-level representations and architectures for event-based object classification and 2. The absence of large real-world event-based datasets. In this paper we address both problems. First, we introduce a novel event-based feature representation together with a new machine learning architecture. Compared to previous approaches, we use local memory units to efficiently leverage past temporal information and build a robust event-based representation. Second, we release the first large real-world event-based dataset for object classification. We compare our method to the state-of-the-art with extensive experiments, showing better classification performance and real-time computation.

The characteristic of the famous existing datasets.

As can be seen in Table {{table:43af4723-b466-4acf-a9d4-94aea48c2ae5}} , compared to the traditional frame-based datasets, event-based datasets are smaller in scale, and the DVS-Gesture has only 1342 samples for 11 categories. Such small and sparse datasets can easily lead to overfitting and unstable convergence, whether for artificial neural networks (ANNs) or spiking neural networks (SNNs). Moreover, it has restricted the development of the event-based algorithm. An intuitive idea is to collect more event-based data. However, due to the scarcity of event cameras, the cost is expensive compared to collecting traditional frame-based datasets. An alternative approach is to apply the data augmentation to the existing datasets. The data augmentation approach improves the quality and quantity of the training data by adding prior knowledge such as rotation and flipping to generate more different representations of the training data. Researchers have proposed many strategies {{cite:7e3a883}}, {{cite:efada09}}, {{cite:c09bb2c}}, {{cite:ad07a1a}} for the traditional image data.


Convolutional 2D Knowledge Graph Embeddings

Abstract: Link prediction for knowledge graphs is the task of predicting missing relationships between entities. Previous work on link prediction has focused on shallow, fast models which can scale to large knowledge graphs. However, these models learn less expressive features than deep, multi-layer models — which potentially limits performance. In this work we introduce ConvE, a multi-layer convolutional network model for link prediction, and report state-of-the-art results for several established datasets. We also show that the model is highly parameter efficient, yielding the same performance as DistMult and R-GCN with 8x and 17x fewer parameters. Analysis of our model suggests that it is particularly effective at modelling nodes with high indegree — which are common in highly-connected, complex knowledge graphs such as Freebase and YAGO3. In addition, it has been noted that the WN18 and FB15k datasets suffer from test set leakage, due to inverse relations from the training set being present in the test set — however, the extent of this issue has so far not been quantified. We find this problem to be severe: a simple rule-based model can achieve state-of-the-art results on both WN18 and FB15k. To ensure that models are evaluated on datasets where simply exploiting inverse relations cannot yield competitive results, we investigate and validate several commonly used datasets — deriving robust variants where necessary. We then perform experiments on these robust datasets for our own and several previously proposed models, and find that ConvE achieves state-of-the-art Mean Reciprocal Rank across all datasets.

Open Graph Benchmark: Datasets for Machine Learning on Graphs

Abstract: We present the Open Graph Benchmark (OGB), a diverse set of challenging and realistic benchmark datasets to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale (up to 100+ million nodes and 1+ billion edges), encompass multiple important graph ML tasks, and cover a diverse range of domains, ranging from social and information networks to biological networks, molecular graphs, source code ASTs, and knowledge graphs. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics. In addition to building the datasets, we also perform extensive benchmark experiments for each dataset. Our experiments suggest that OGB datasets present significant challenges of scalability to large-scale graphs and out-of-distribution generalization under realistic data splits, indicating fruitful opportunities for future research. Finally, OGB provides an automated end-to-end graph ML pipeline that simplifies and standardizes the process of graph data loading, experimental setup, and model evaluation. OGB will be regularly updated and welcomes inputs from the community. OGB datasets as well as data loaders, evaluation scripts, baseline code, and leaderboards are publicly available at this https URL .

Statistics of the KG completion datasets.

Dataset statistics.
We summarize the statistics of different
benchmark datasets
in Table {{table:0b853efc-b77a-4620-96a9-a773fb620231}} .
As shown,
ogbl-biokg and ogbl-wikikg2
have much larger size compared with
WN18RR and FB15k-237.


Weighted Low-rank Tensor Recovery for Hyperspectral Image Restoration

Abstract: Hyperspectral imaging, providing abundant spatial and spectral information simultaneously, has attracted a lot of interest in recent years. Unfortunately, due to the hardware limitations, the hyperspectral image (HSI) is vulnerable to various degradations, such noises (random noise, HSI denoising), blurs (Gaussian and uniform blur, HSI deblurring), and down-sampled (both spectral and spatial downsample, HSI super-resolution). Previous HSI restoration methods are designed for one specific task only. Besides, most of them start from the 1-D vector or 2-D matrix models and cannot fully exploit the structurally spectral-spatial correlation in 3-D HSI. To overcome these limitations, in this work, we propose a unified low-rank tensor recovery model for comprehensive HSI restoration tasks, in which non-local similarity between spectral-spatial cubic and spectral correlation are simultaneously captured by 3-order tensors. Further, to improve the capability and flexibility, we formulate it as a weighted low-rank tensor recovery (WLRTR) model by treating the singular values differently, and study its analytical solution. We also consider the exclusive stripe noise in HSI as the gross error by extending WLRTR to robust principal component analysis (WLRTR-RPCA). Extensive experiments demonstrate the proposed WLRTR models consistently outperform state-of-the-arts in typical low level vision HSI tasks, including denoising, destriping, deblurring and super-resolution.

Hyperspectral Image Restoration via Total Variation Regularized Low-rank Tensor Decomposition

Abstract: Hyperspectral images (HSIs) are often corrupted by a mixture of several types of noise during the acquisition process, e.g., Gaussian noise, impulse noise, dead lines, stripes, and many others. Such complex noise could degrade the quality of the acquired HSIs, limiting the precision of the subsequent processing. In this paper, we present a novel tensor-based HSI restoration approach by fully identifying the intrinsic structures of the clean HSI part and the mixed noise part respectively. Specifically, for the clean HSI part, we use tensor Tucker decomposition to describe the global correlation among all bands, and an anisotropic spatial-spectral total variation (SSTV) regularization to characterize the piecewise smooth structure in both spatial and spectral domains. For the mixed noise part, we adopt the $\ell_1$ norm regularization to detect the sparse noise, including stripes, impulse noise, and dead pixels. Despite that TV regulariztion has the ability of removing Gaussian noise, the Frobenius norm term is further used to model heavy Gaussian noise for some real-world scenarios. Then, we develop an efficient algorithm for solving the resulting optimization problem by using the augmented Lagrange multiplier (ALM) method. Finally, extensive experiments on simulated and real-world noise HSIs are carried out to demonstrate the superiority of the proposed method over the existing state-of-the-art ones.

Constraints on strips and image of typical LRTA-based HSI destriping methods

Technically, there are two ways to treat stripes in HSIs. Generally, stripes are assumed to follow a specific distribution that is different with the distribution of HSIs, such that stripes are separated from image information by image decomposition paradigm {{cite:86079de}}, {{cite:db47165}}. On the one hand, when the density of stripes is sparse, tensorial robust principal component analysis (TRPCA) is extended by considering stripes as sparse noise while LR tensor prior is utilized to model image component {{cite:5bf2ce7}}, {{cite:68cc0dd}}. However, on the other hand, according to imaging mechanism of spectrometers, the distribution of stripes in an HSI is unpredictable, which tends to be heavy in some cases and cannot be modeled by simple sparsity assumption {{cite:db47165}}. Therefore, some researches have been focusing on exploring extra priors of strips distribution. The researches along this direction hold the view that stripes have low-rankness and group sparsity {{cite:128dba4}}, {{cite:76e9df6}}, {{cite:db47165}}. Besides, the stripes are also regarded as Gaussian error and the stripe removal issues are also naturally resolved by solving the mixed denoising problem {{cite:9f44b88}}, {{cite:ce900d1}}, {{cite:0c28eb3}}. Table {{table:5a932d1c-aec4-4324-91d9-548c0bb0fbcb}}  lists the main LRTA-based HSI destriping methods based on image decomposition.


Convolutional Neural Networks Applied to House Numbers Digit Classification

Abstract: We classify digits of real-world house numbers using convolutional neural networks (ConvNets). ConvNets are hierarchical feature learning neural networks whose structure is biologically inspired. Unlike many popular vision approaches that are hand-designed, ConvNets can automatically learn a unique set of features optimized for a given task. We augmented the traditional ConvNet architecture by learning multi-stage features and by using Lp pooling and establish a new state-of-the-art of 94.85% accuracy on the SVHN dataset (45.2% error improvement). Furthermore, we analyze the benefits of different pooling methods and multi-stage features in ConvNets. The source code and a tutorial are available at eblearn.sf.net.

Playing for Data: Ground Truth from Computer Games

Abstract: Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set.

PubLayNet: largest dataset ever for document layout analysis

Abstract: Recognizing the layout of unstructured digital documents is an important step when parsing the documents into structured machine-readable format for downstream applications. Deep neural networks that are developed for computer vision have been proven to be an effective method to analyze layout of document images. However, document layout datasets that are currently publicly available are several magnitudes smaller than established computing vision datasets. Models have to be trained by transfer learning from a base model that is pre-trained on a traditional computer vision dataset. In this paper, we develop the PubLayNet dataset for document layout analysis by automatically matching the XML representations and the content of over 1 million PDF articles that are publicly available on PubMed Central. The size of the dataset is comparable to established computer vision datasets, containing over 360 thousand document images, where typical document layout elements are annotated. The experiments demonstrate that deep neural networks trained on PubLayNet accurately recognize the layout of scientific articles. The pre-trained models are also a more effective base mode for transfer learning on a different document domain. We release the dataset (https://github.com/ibm-aur-nlp/PubLayNet) to support development and evaluation of more advanced models for document layout analysis.

Tasks and Datasets: The source and target datasets are listed in the first two columns. The fourth column indicates whether there is a shift in label space while adapting from the source to the target domain.

We succinctly present the tasks, datasets, and evaluation metrics in Table {{table:c465a66e-3c23-48e3-8812-e72b46e590b3}} . Under classification settings, DistillAdapt, even without access to annotated source data, performs similar to or better than (with a variance of {{formula:ecdf878b-fc7b-421a-b9d7-442e3ea9bd87}} % in accuracy) prior active domain adaptation methods {{cite:2aae68e}}, {{cite:2d328fd}} that use large amounts (more than {{formula:2ad7ca0d-9a78-4cb7-a9bc-edc71c5c0c71}}  samples) of annotated source data. Furthermore, we conduct experiments on MNIST under 2 distinct cases of shift in output label space, and show that DistillAdapt can achieve atleast {{formula:7fbfddea-4247-4b25-a409-1f6f0381c693}} % of the accuracy obtained when there is no shift in label space. Thus, DistillAdapt can handle shifts in label space. Our experiments on CityScapes for semantic segmentation at various budgets reveal an improvement of atleast {{formula:f648a1fc-7724-41f3-b755-c87d6b2f952e}} % over source-free fine-tuning (i.e. training the model without {{formula:a97efcc1-5138-4bd1-94c3-dd65c02f93bb}} ). Finally, we conduct experiments on document layout adaptation from PubLayNet to DSSE where there is a shift in output label space, DistillAdapt imparts a relative improvement of {{formula:6e14e969-5278-452f-88e6-64640bea15e0}} % over fine-tuning (i.e. training the model without {{formula:0e1a3e5a-4d22-4994-b38d-04d1741f5603}} ).


GRAB: A Dataset of Whole-Body Human Grasping of Objects

Abstract: Training computers to understand, model, and synthesize human grasping requires a rich dataset containing complex 3D object shapes, detailed contact information, hand pose and shape, and the 3D body motion over time. While "grasping" is commonly thought of as a single hand stably lifting an object, we capture the motion of the entire body and adopt the generalized notion of "whole-body grasps". Thus, we collect a new dataset, called GRAB (GRasping Actions with Bodies), of whole-body grasps, containing full 3D shape and pose sequences of 10 subjects interacting with 51 everyday objects of varying shape and size. Given MoCap markers, we fit the full 3D body shape and pose, including the articulated face and hands, as well as the 3D object pose. This gives detailed 3D meshes over time, from which we compute contact between the body and object. This is a unique dataset, that goes well beyond existing ones for modeling and understanding how humans grasp and manipulate objects, how their full body is involved, and how interaction varies with the task. We illustrate the practical value of GRAB with an example application; we train GrabNet, a conditional generative network, to predict 3D hand grasps for unseen 3D object shapes. The dataset and code are available for research purposes at https://grab.is.tue.mpg.de.

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Abstract: Recent approaches in depth-based human activity analysis achieved outstanding performance and proved the effectiveness of 3D representation for classification of action classes. Currently available depth-based and RGB+Dbased action recognition benchmarks have a number of limitations, including the lack of training samples, distinct class labels, camera views and variety of subjects. In this paper we introduce a large-scale dataset for RGB+D human action recognition with more than 56 thousand video samples and 4 million frames, collected from 40 distinct subjects. Our dataset contains 60 different action classes including daily, mutual, and health-related actions. In addition, we propose a new recurrent neural network structure to model the long-term temporal correlation of the features for each body part, and utilize them for better action classification. Experimental results show the advantages of applying deep learning methods over state-of-the-art handcrafted features on the suggested cross-subject and cross-view evaluation criteria for our dataset. The introduction of this large scale dataset will enable the community to apply, develop and adapt various data-hungry learning techniques for the task of depth-based and RGB+D-based human activity analysis.

BABEL: Bodies, Action and Behavior with English Labels

Abstract: Understanding the semantics of human movement – the what, how and why of the movement – is an important problem that requires datasets of human actions with semantic labels. Existing datasets take one of two approaches. Large-scale video datasets contain many action labels but do not contain ground-truth 3D human motion. Alternatively, motion-capture (mocap) datasets have precise body motions but are limited to a small number of actions. To address this, we present BABEL, a large dataset with language labels describing the actions being performed in mocap sequences. BABEL consists of language labels for over 43 hours of mocap sequences from AMASS, containing over 250 unique actions. Each action label in BABEL is precisely aligned with the duration of the corresponding action in the mocap sequence. BABELalso allows overlap of multiple actions, that may each span different durations. This results in a total of over 66000 action segments. The dense annotations can be leveraged for tasks like action recognition, temporal localization, motion synthesis, etc. To demonstrate the value of BABEL as a benchmark, we evaluate the performance of models on 3D action recognition. We demonstrate that BABEL poses interesting learning challenges that are applicable to real-world scenarios, and can serve as a useful benchmark for progress in 3D action recognition. The dataset, baseline methods, and evaluation code are available and supported for academic research purposes at https://babel.is.tue.mpg.de/.

Datasets' details. We list the range of motion length in frames, the number of training/testing samples, the number of training samples with action transitions and the number of actions in each dataset.

We evaluate our method on three different datasets. Each motion sequence in these datasets is annotated with a single action label, except for BABEL {{cite:075b427}}. Some information for each dataset is provided in Table {{table:206ce7f7-4db4-4dca-b32b-45c21bfd2113}} . We also evaluate on the dataset HumanAct12 {{cite:504f353}}. The results are in the supplementary material.

Finally, we compare the performance of using the ground-truth transitions (“gt-transi”) to that of our weakly-supervised strategy (“weakly-sup”) on BABEL in Table {{table:ac89efd0-8394-47fa-8c6a-34d9f4e6a2c8}} . Since the limited ground-truth transitions in BABEL do not cover all possible cases, using them as supervision is ineffective. Specifically, as shown in Table {{table:206ce7f7-4db4-4dca-b32b-45c21bfd2113}} , there are only around 2500 ground-truth transition sequences, depicting 170 types of transitions. By contrast, our weakly-supervised strategy leverages almost 100,000 pseudo transitions covering all 380 possible types. This further evidences the importance of our weakly-supervised action transition learning strategy.


Pluralistic Image Completion

Abstract: Most image completion methods produce only one result for each masked input, although there may be many reasonable possibilities. In this paper, we present an approach for pluralistic image completion – the task of generating multiple and diverse plausible solutions for image completion. A major challenge faced by learning-based approaches is that usually only one ground truth training instance per label. As such, sampling from conditional VAEs still leads to minimal diversity. To overcome this, we propose a novel and probabilistically principled framework with two parallel paths. One is a reconstructive path that utilizes the only one given ground truth to get prior distribution of missing parts and rebuild the original image from this distribution. The other is a generative path for which the conditional prior is coupled to the distribution obtained in the reconstructive path. Both are supported by GANs. We also introduce a new short+long term attention layer that exploits distant relations among decoder and encoder features, improving appearance consistency. When tested on datasets with buildings (Paris), faces (CelebA-HQ), and natural images (ImageNet), our method not only generated higherquality completion results, but also with multiple and diverse plausible outputs.

Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE

Abstract: Given an incomplete image without additional constraint, image inpainting natively allows for multiple solutions as long as they appear plausible. Recently, multiple-solution inpainting methods have been proposed and shown the potential of generating diverse results. However, these methods have difficulty in ensuring the quality of each solution, e.g. they produce distorted structure and/or blurry texture. We propose a two-stage model for diverse inpainting, where the first stage generates multiple coarse results each of which has a different structure, and the second stage refines each coarse result separately by augmenting texture. The proposed model is inspired by the hierarchical vector quantized variational auto-encoder (VQ-VAE), whose hierarchical architecture disentangles structural and textural information. In addition, the vector quantization in VQVAE enables autoregressive modeling of the discrete distribution over the structural information. Sampling from the distribution can easily generate diverse and high-quality structures, making up the first stage of our model. In the second stage, we propose a structural attention module inside the texture generation network, where the module utilizes the structural information to capture distant correlations. We further reuse the VQ-VAE to calculate two feature losses, which help improve structure coherence and texture realism, respectively. Experimental results on CelebA-HQ, Places2, and ImageNet datasets show that our method not only enhances the diversity of the inpainting solutions but also improves the visual quality of the generated multiple images. Code and models are available at: https://github.com/USTC-JialunPeng/Diverse-Structure-Inpainting.

High-Fidelity Pluralistic Image Completion with Transformers

Abstract: Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This paper brings the best of both worlds to pluralistic image completion: appearance prior reconstruction with transformer and texture replenishment with CNN. The former transformer recovers pluralistic coherent structures together with some coarse textures, while the latter CNN enhances the local texture details of coarse priors guided by the high-resolution masked images. The proposed method vastly outperforms state-of-the-art methods in terms of three aspects: 1) large performance boost on image fidelity even compared to deterministic completion methods; 2) better diversity and higher fidelity for pluralistic completion; 3) exceptional generalization ability on large masks and generic dataset, like ImageNet.

PD-GAN: Probabilistic Diverse GAN for Image Inpainting

Abstract: We propose PD-GAN, a probabilistic diverse GAN for image inpainting. Given an input image with arbitrary hole regions, PD-GAN produces multiple inpainting results with diverse and visually realistic content. Our PD-GAN is built upon a vanilla GAN which generates images based on random noise. During image generation, we modulate deep features of input random noise from coarse-to-fine by injecting an initially restored image and the hole regions in multiple scales. We argue that during hole filling, the pixels near the hole boundary should be more deterministic (i.e., with higher probability trusting the context and initially restored image to create natural inpainting boundary), while those pixels lie in the center of the hole should enjoy more degrees of freedom (i.e., more likely to depend on the random noise for enhancing diversity). To this end, we propose spatially probabilistic diversity normalization (SPDNorm) inside the modulation to model the probability of generating a pixel conditioned on the context information. SPDNorm dynamically balances the realism and diversity inside the hole region, making the generated content more diverse towards the hole center and resemble neighboring image content more towards the hole boundary. Meanwhile, we propose a perceptual diversity loss to further empower PD-GAN for diverse content generation. Experiments on benchmark datasets including CelebA-HQ, Places2 and Paris Street View indicate that PD-GAN is effective for diverse and visually realistic image restoration.

Diverse Image Inpainting with Bidirectional and Autoregressive Transformers

Abstract: Image inpainting is an underdetermined inverse problem, which naturally allows diverse contents to fill up the missing or corrupted regions realistically. Prevalent approaches using convolutional neural networks (CNNs) can synthesize visually pleasant contents, but CNNs suffer from limited perception fields for capturing global features. With image-level attention, transformers enable to model long-range dependencies and generate diverse contents with autoregressive modeling of pixel-sequence distributions. However, the unidirectional attention in autoregressive transformers is suboptimal as corrupted image regions may have arbitrary shapes with contexts from any direction. We propose BAT-Fill, an innovative image inpainting framework that introduces a novel bidirectional autoregressive transformer (BAT) for image inpainting. BAT utilizes the transformers to learn autoregressive distributions, which naturally allows the diverse generation of missing contents. In addition, it incorporates the masked language model like BERT, which enables bidirectionally modeling of contextual information of missing regions for better image completion. Extensive experiments over multiple datasets show that BAT-Fill achieves superior diversity and fidelity in image inpainting qualitatively and quantitatively.

Generative methods used in the analyzed state-of-the-art proposals for diverse inpainting.

How to Achieve Multiple and Diverse Inpainting Results?

In this section, we will analyze the different families of generative models proposed in the literature to realize diverse image inpainting. We will in particular describe generative adversarial networks (GAN), variational autoencoders (VAE), autoregressive models and transformers. We will also detail the different objective losses proposed to train these networks. Finally, for each family of models, we will review several state-of-the-art diverse inpainting methods that relate to this model. Table {{table:605a2d27-061a-4576-832f-d7804774abc0}}  lists all the methods that will be reviewed in this section.


Reconfigurable and approximate computing for video coding

Abstract: The Chapter begins with a discussion of the constraints and needs of video coding systems. The lack in flexibility of traditional monolithic codec specifications, not suitable to model commonalities among codecs and foster reusability among successive codec generations/updates, was the main trigger for the development of a new standard initiative within the ISO/IEC MPEG committee, called reconfigurable video coding (RVC). The MPEG-RVC framework exploits the dataflow nature behind video coding to foster flexible and reconfigurable codec design, as well as to support dynamic reconfiguration. The Chapter goes on to consider that the inherent resiliency of various functional blocks (like motion estimation in the high-efficiency video coding, HEVC) and the varying levels of user perception make video coding suitable to apply approximate computing techniques. Approximate computing, if properly supported at design time, allows achieving run-time trade-offs, representing a new direction in hardware-software codesign research. The main assumption behind approximate computing, exploited within video coding, is that the degree of accuracy (in this case during codec execution) is not required to be the same all the time. The final part of the Chapter attempts to put together the concepts addressed and remarks on which are, in the authors' opinion, some interesting research directions.

Potentials of reconfigurable approximate multi-frequency architectures with respect to other literature works. H, M and L are the system operating modes, while in brackets are reported results achievable when multi-frequency is enabled.

To understand the potentials of approximate multi-frequency approaches in the practice, Table {{table:938a6866-f657-466a-901e-12bba105e062}}  reports a comparison of the work proposed in {{cite:d8670ec}} with similar state of the art solutions. The detailed discussion of such results is beyond the scope of this chapter and it is available in {{cite:d8670ec}}. However, from the table it is possible to appreciate how: i) the power/energy versus quality trade-off curve exists despite the additional logic required to implement reconfiguration on hardware; ii) the reconfigurable approximate multi-frequency solution is capable of competing and beating most of the literature, even not dynamic (not reconfigurable) approaches in terms of throughput and energy efficiency.