News, Social Media, E-Commerce, Dialogue

General, News, Social Media, Dialogue, E-Commerce

18 language Pairs, including Japanese, English and Polish

English, Arabic, Chinese, German, Spanish

English, Chinese, Japanese, German, French

Linguistically Informed ChatGPT Prompts to Enhance Japanese-Chinese Machine Translation: A Case Study on Attributive Clauses

Abstract: In the field of Japanese-Chinese translation linguistics, the issue of correctly translating attributive clauses has persistently proven to be challenging. Present-day machine translation tools often fail to accurately translate attributive clauses from Japanese to Chinese. In light of this, this paper investigates the linguistic problem underlying such difficulties, namely how does the semantic role of the modified noun affect the selection of translation patterns for attributive clauses, from a linguistic perspective. To ad-dress these difficulties, a pre-edit scheme is proposed, which aims to enhance the accuracy of translation. Furthermore, we propose a novel two-step prompt strategy, which combines this pre-edit scheme with ChatGPT, currently the most widely used large language model. This prompt strategy is capable of optimizing translation input in zero-shot scenarios and has been demonstrated to improve the average translation accuracy score by over 35%.

Towards Making the Most of ChatGPT for Machine Translation

Abstract: ChatGPT shows remarkable capabilities for machine translation (MT). Several prior studies have shown that it achieves comparable results to commercial systems for high-resource languages, but lags behind in complex tasks, e.g., low-resource and distant-language-pairs translation. However, they usually adopt simple prompts which can not fully elicit the capability of ChatGPT. In this paper, we aim to further mine ChatGPT's translation ability by revisiting several aspects: temperature, task information, and domain information, and correspondingly propose an optimal temperature setting and two (simple but effective) prompts: Task-Specific Prompts (TSP) and Domain-Specific Prompts (DSP). We show that: 1) The performance of ChatGPT depends largely on temperature, and a lower temperature usually can achieve better performance; 2) Emphasizing the task information can further improve ChatGPT's performance, particularly in complex MT tasks; 3) Introducing domain information can elicit ChatGPT's generalization ability and improve its performance in the specific domain; 4) ChatGPT tends to generate hallucinations for non-English-centric MT tasks, which can be partially addressed by our proposed prompts but still need to be highlighted for the MT/NLP community. We also explore the effects of advanced in-context learning strategies and find a (negative but interesting) observation: the powerful chain-of-thought prompt leads to word-by-word translation behavior, thus bringing significant translation degradation.

Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine

Abstract: This report provides a preliminary evaluation of ChatGPT for machine translation, including translation prompt, multilingual translation, and translation robustness. We adopt the prompts advised by ChatGPT to trigger its translation ability and find that the candidate prompts generally work well with minor performance differences. By evaluating on a number of benchmark test sets, we find that ChatGPT performs competitively with commercial translation products (e.g., Google Translate) on high-resource European languages but lags behind significantly on low-resource or distant languages. As for the translation robustness, ChatGPT does not perform as well as the commercial systems on biomedical abstracts or Reddit comments but exhibits good results on spoken language. Further, we explore an interesting strategy named $\mathbf{pivot~prompting}$ for distant languages, which asks ChatGPT to translate the source sentence into a high-resource pivot language before into the target language, improving the translation performance noticeably. With the launch of the GPT-4 engine, the translation performance of ChatGPT is significantly boosted, becoming comparable to commercial translation products, even for distant languages. Human analysis on Google Translate and ChatGPT suggests that ChatGPT with GPT-3.5 tends to generate more hallucinations and mis-translation errors while that with GPT-4 makes the least errors. In other words, ChatGPT has already become a good translator. Please refer to our Github project for more details: https://github.com/wxjiao/Is-ChatGPT-A-Good-Translator

How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation

Abstract: Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.

How to Design Translation Prompts for ChatGPT: An Empirical Study

Abstract: The recently released ChatGPT has demonstrated surprising abilities in natural language understanding and natural language generation. Machine translation relies heavily on the abilities of language understanding and generation. Thus, in this paper, we explore how to assist machine translation with ChatGPT. We adopt several translation prompts on a wide range of translations. Our experimental results show that ChatGPT with designed translation prompts can achieve comparable or better performance over commercial translation systems for high-resource language translations. We further evaluate the translation quality using multiple references, and ChatGPT achieves superior performance compared to commercial systems. We also conduct experiments on domain-specific translations, the final results show that ChatGPT is able to comprehend the provided domain keyword and adjust accordingly to output proper translations. At last, we perform few-shot prompts that show consistent improvement across different base prompts. Our work provides empirical evidence that ChatGPT still has great potential in translations.

Document-Level Machine Translation with Large Language Models

Abstract: Large language models (LLMs) such as ChatGPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking document-level machine translation (MT) as a testbed, this paper provides an in-depth evaluation of LLMs' ability on discourse modeling. The study focuses on three aspects: 1) Effects of Context-Aware Prompts, where we investigate the impact of different prompts on document-level translation quality and discourse phenomena; 2) Comparison of Translation Models, where we compare the translation performance of ChatGPT with commercial MT systems and advanced document-level MT methods; 3) Analysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and shed light on impacts of training techniques on discourse modeling. By evaluating on a number of benchmarks, we surprisingly find that LLMs have demonstrated superior performance and show potential to become a new paradigm for document-level translation: 1) leveraging their powerful long-text modeling capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms of human evaluation; 2) GPT-4 demonstrates a stronger ability for probing linguistic knowledge than GPT-3.5. This work highlights the challenges and opportunities of LLMs for MT, which we hope can inspire the future design and evaluation of LLMs.We release our data and annotations at https://github.com/longyuewangdcu/Document-MT-LLM.

Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis

Abstract: Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT). In this paper, we systematically investigate the advantages and challenges of LLMs for MMT by answering two questions: 1) How well do LLMs perform in translating massive languages? 2) Which factors affect LLMs' performance in translation? We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4. Our empirical results show that translation capabilities of LLMs are continually improving. GPT-4 has beat the strong supervised baseline NLLB in 40.91% of translation directions but still faces a large gap towards the commercial translation system, especially on low-resource languages. Through further analysis, we discover that LLMs exhibit new working patterns when used for MMT. First, instruction semantics can surprisingly be ignored when given in-context exemplars. Second, cross-lingual exemplars can provide better task guidance for low-resource translation than exemplars in the same language pairs. Third, LLM can acquire translation ability in a resource-efficient way and generate moderate translation even on zero-resource languages.

Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist

Abstract: Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets. However, their ability to translate paragraphs and documents remains unexplored because evaluation in these settings is costly and difficult. We show through a rigorous human evaluation that asking the GPT-3.5 (text-davinci-003) LLM to translate an entire literary paragraph (e.g., from a novel) at once results in higher-quality translations than standard sentence-by-sentence translation across 18 linguistically-diverse language pairs (e.g., translating into and out of Japanese, Polish, and English). Our evaluation, which took approximately 350 hours of effort for annotation and analysis, is conducted by hiring translators fluent in both the source and target language and asking them to provide both span-level error annotations as well as preference judgments of which system’s translations are better. We observe that discourse-level LLM translators commit fewer mistranslations, grammar errors, and stylistic inconsistencies than sentence-level approaches. With that said, critical errors still abound, including occasional content omissions, and a human translator’s intervention remains necessary to ensure that the author’s voice remains intact. We publicly release our dataset and error annotations to spur future research on the evaluation of document-level literary translation.

Adaptive Machine Translation with Large Language Models

Abstract: Consistency is a key requirement of high-quality translation. It is especially important to adhere to pre-approved terminology and adapt to corrected translations in domain-specific projects. Machine translation (MT) has achieved significant progress in the area of domain adaptation. However, real-time adaptation remains challenging. Large-scale language models (LLMs) have recently shown interesting capabilities of in-context learning, where they learn to replicate certain input-output text generation patterns, without further fine-tuning. By feeding an LLM at inference time with a prompt that consists of a list of translation pairs, it can then simulate the domain and style characteristics. This work aims to investigate how we can utilize in-context learning to improve real-time adaptive MT. Our extensive experiments show promising results at translation time. For example, GPT-3.5 can adapt to a set of in-domain sentence pairs and/or terminology while translating a new sentence. We observe that the translation quality with few-shot in-context learning can surpass that of strong encoder-decoder MT systems, especially for high-resource languages. Moreover, we investigate whether we can combine MT from strong encoder-decoder models with fuzzy matches, which can further improve translation quality, especially for less supported languages. We conduct our experiments across five diverse language pairs, namely English-to-Arabic (EN-AR), English-to-Chinese (EN-ZH), English-to-French (EN-FR), English-to-Kinyarwanda (EN-RW), and English-to-Spanish (EN-ES).

Exploring Human-Like Translation Strategy with Large Language Models

Abstract: Abstract Large language models (LLMs) have demonstrated impressive capabilities in general scenarios, exhibiting a level of aptitude that approaches, in some aspects even surpasses, human-level intelligence. Among their numerous skills, the translation abilities of LLMs have received considerable attention. Compared to typical machine translation that focuses solely on source-to-target mapping, LLM-based translation can potentially mimic the human translation process, which might take preparatory steps to ensure high-quality translation. This work explores this possibility by proposing the MAPS framework, which stands for Multi-Aspect Prompting and Selection. Specifically, we enable LLMs first to analyze the given source sentence and induce three aspects of translation-related knowledge (keywords, topics, and relevant demonstrations) to guide the final translation process. Moreover, we employ a selection mechanism based on quality estimation to filter out noisy and unhelpful knowledge. Both automatic (3 LLMs × 11 directions × 2 automatic metrics) and human evaluation (preference study and MQM) demonstrate the effectiveness of MAPS. Further analysis shows that by mimicking the human translation process, MAPS reduces various translation errors such as hallucination, ambiguity, mistranslation, awkward style, untranslated text, and omission. Source code is available at https://github.com/zwhe99/MAPS-mt.

Leveraging GPT-4 for Automatic Translation Post-Editing

Abstract: While Neural Machine Translation (NMT) represents the leading approach to Machine Translation (MT), the outputs of NMT models still require translation post-editing to rectify errors and enhance quality under critical settings. In this work, we formalize the task of direct translation post-editing with Large Language Models (LLMs) and explore the use of GPT-4 to automatically post-edit NMT outputs across several language pairs. Our results demonstrate that GPT-4 is adept at translation post-editing, producing meaningful and trustworthy edits to translations that help improve its general quality as well as remove different classes of major errors in translations. In particular, human evaluations on assessing edit trustworthiness show that GPT-4 exhibits a large improvement over the prior state-of-the-art LLM. Notably, we improve upon state-of-the-art performance on WMT-22 English-Chinese, English-German, Chinese-English and German-English language pairs using GPT-4 based post-editing, as evaluated by state-of-the-art MT quality metrics. However, we also show that GPT-4 could produce hallucinated edits, thereby urging caution in its use as an expert translation post-editor.

Do GPTs Produce Less Literal Translations?

Abstract: Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose language models capable of addressing many natural language generation or understanding tasks. On the task of Machine Translation (MT), multiple works have investigated few-shot prompting mechanisms to elicit better translations from LLMs. However, there has been relatively little investigation on how such translations differ qualitatively from the translations generated by standard Neural Machine Translation (NMT) models. In this work, we investigate these differences in terms of the literalness of translations produced by the two systems. Using literalness measures involving word alignment and monotonicity, we find that translations out of English (E-X) from GPTs tend to be less literal, while exhibiting similar or better scores on MT quality metrics. We demonstrate that this finding is borne out in human evaluations as well. We then show that these differences are especially pronounced when translating sentences that contain idiomatic expressions.

Summary of research works exploring GLLMs for machine translation. Here ZS represents zero-shot, and FS represents few-shot.

Research works exploring GLLMs for machine translation. In recent times, GLLMs like ChatGPT and GPT-4 demonstrated remarkable performances in both natural language understanding and generation tasks. A good machine translation system requires strong natural language understanding and generation skills. As ChatGPT and GPT-4 possess strong natural language understanding and generation skills, the research community investigated the effectiveness of these models for machine translation across various domains like news {{cite:31a9590}}, {{cite:534b40d}}, {{cite:0cb646f}}, {{cite:0d17d56}}, healthcare {{cite:31a9590}}, {{cite:5e14c09}}, social media {{cite:5e14c09}}, {{cite:534b40d}}, {{cite:0cb646f}}, {{cite:0d17d56}}, dialogue {{cite:534b40d}}, {{cite:0d17d56}}, {{cite:0cb646f}} and e-commerce {{cite:534b40d}}, {{cite:0cb646f}}. Most of the research works focused on sentence-level machine translation {{cite:64c6771}}, {{cite:31a9590}}, {{cite:5e14c09}}, {{cite:534b40d}}, {{cite:0cb646f}}, {{cite:c08892a}}, {{cite:f43ce11}}, {{cite:4cca9bc}}, {{cite:0759d4e}}, {{cite:08257f9}}, {{cite:b88d986}}, {{cite:3db8377}}, except a few research works focused on paragraph-level machine translation {{cite:f83e9dd}}, {{cite:4cca9bc}} and document-level machine translation {{cite:534b40d}}, {{cite:0d17d56}}. As advanced prompting methods allow GLLMs to perform well, some of the research works investigated the effectiveness of advanced prompting strategies like pivot {{cite:5e14c09}}, chain-of-thought {{cite:b88d986}} and multi-aspect prompting and selection {{cite:08257f9}}. Table {{table:15fc4849-d9f8-4871-be91-1f3bc5a4d87f}}  presents a summary of research works exploring GLLMs for machine translation across various domains and languages.


Scikit-learn: Machine Learning in Python

Abstract: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.

Network Inference via the Time-Varying Graphical Lasso

Abstract: Many important problems can be modeled as a system of interconnected entities, where each entity is recording time-dependent observations or measurements. In order to spot trends, detect anomalies, and interpret the temporal dynamics of such data, it is essential to understand the relationships between the different entities and how these relationships evolve over time. In this paper, we introduce the time-varying graphical lasso (TVGL), a method of inferring time-varying networks from raw time series data. We cast the problem in terms of estimating a sparse time-varying inverse covariance matrix, which reveals a dynamic network of interdependencies between the entities. Since dynamic network inference is a computationally expensive task, we derive a scalable message-passing algorithm based on the Alternating Direction Method of Multipliers (ADMM) to solve this problem in an efficient way. We also discuss several extensions, including a streaming algorithm to update the model and incorporate new observations in real time. Finally, we evaluate our TVGL algorithm on both real and synthetic datasets, obtaining interpretable results and outperforming state-of-the-art baselines in terms of both accuracy and scalability.

Iterative Thresholding Algorithm for Sparse Inverse Covariance Estimation

Abstract: The L1-regularized maximum likelihood estimation problem has recently become a topic of great interest within the machine learning, statistics, and optimization communities as a method for producing sparse inverse covariance estimators. In this paper, a proximal gradient method (G-ISTA) for performing L1-regularized covariance matrix estimation is presented. Although numerous algorithms have been proposed for solving this problem, this simple proximal gradient method is found to have attractive theoretical and numerical properties. G-ISTA has a linear rate of convergence, resulting in an O(log e) iteration complexity to reach a tolerance of e. This paper gives eigenvalue bounds for the G-ISTA iterates, providing a closed-form linear convergence rate. The rate is shown to be closely related to the condition number of the optimal point. Numerical convergence results and timing comparisons for the proposed method are presented. G-ISTA is shown to perform very well, especially when the optimal point is well-conditioned.

The joint graphical lasso for inverse covariance estimation across multiple classes

Abstract: We consider the problem of estimating multiple related Gaussian graphical models from a high dimensional data set with observations belonging to distinct classes. We propose the joint graphical lasso, which borrows strength across the classes to estimate multiple graphical models that share certain characteristics, such as the locations or weights of non‐zero edges. Our approach is based on maximizing a penalized log‐likelihood. We employ generalized fused lasso or group lasso penalties and implement a fast alternating directions method of multipliers algorithm to solve the corresponding convex optimization problems. The performance of the method proposed is illustrated through simulated and real data examples.

A Fast and Scalable Joint Estimator for Learning Multiple Related Sparse Gaussian Graphical Models

Abstract: Estimating multiple sparse Gaussian Graphical Models (sGGMs) jointly for many related tasks (large $K$) under a high-dimensional (large $p$) situation is an important task. Most previous studies for the joint estimation of multiple sGGMs rely on penalized log-likelihood estimators that involve expensive and difficult non-smooth optimizations. We propose a novel approach, FASJEM for \underline{fa}st and \underline{s}calable \underline{j}oint structure-\underline{e}stimation of \underline{m}ultiple sGGMs at a large scale. As the first study of joint sGGM using the Elementary Estimator framework, our work has three major contributions: (1) We solve FASJEM through an entry-wise manner which is parallelizable. (2) We choose a proximal algorithm to optimize FASJEM. This improves the computational efficiency from $O(Kp^3)$ to $O(Kp^2)$ and reduces the memory requirement from $O(Kp^2)$ to $O(K)$. (3) We theoretically prove that FASJEM achieves a consistent estimation with a convergence rate of $O(\log(Kp)/n_{tot})$. On several synthetic and four real-world datasets, FASJEM shows significant improvements over baselines on accuracy, computational complexity, and memory costs.

Large-Scale Sparse Inverse Covariance Estimation via Thresholding and Max-Det Matrix Completion

Abstract: The sparse inverse covariance estimation problem is commonly solved using an $\ell_{1}$-regularized Gaussian maximum likelihood estimator known as "graphical lasso", but its computational cost becomes prohibitive for large data sets. A recent line of results showed--under mild assumptions--that the graphical lasso estimator can be retrieved by soft-thresholding the sample covariance matrix and solving a maximum determinant matrix completion (MDMC) problem. This paper proves an extension of this result, and describes a Newton-CG algorithm to efficiently solve the MDMC problem. Assuming that the thresholded sample covariance matrix is sparse with a sparse Cholesky factorization, we prove that the algorithm converges to an $\epsilon$-accurate solution in $O(n\log(1/\epsilon))$ time and $O(n)$ memory. The algorithm is highly efficient in practice: we solve the associated MDMC problems with as many as 200,000 variables to 7-9 digits of accuracy in less than an hour on a standard laptop computer running MATLAB.

The Sylvester Graphical Lasso (SyGlasso)

Abstract: This paper introduces the Sylvester graphical lasso (SyGlasso) that captures multiway dependencies present in tensor-valued data. The model is based on the Sylvester equation that defines a generative model. The proposed model complements the tensor graphical lasso (Greenewald et al., 2019) that imposes a Kronecker sum model for the inverse covariance matrix by providing an alternative Kronecker sum model that is generative and interpretable. A nodewise regression approach is adopted for estimating the conditional independence relationships among variables. The statistical convergence of the method is established, and empirical studies are provided to demonstrate the recovery of meaningful conditional dependency graphs. We apply the SyGlasso to an electroencephalography (EEG) study to compare the brain connectivity of alcoholic and nonalcoholic subjects. We demonstrate that our model can simultaneously estimate both the brain connectivity and its temporal dependencies.

GLAD: Learning Sparse Graph Recovery

Abstract: Recovering sparse conditional independence graphs from data is a fundamental problem in machine learning with wide applications. A popular formulation of the problem is an $\ell_1$ regularized maximum likelihood estimation. Many convex optimization algorithms have been designed to solve this formulation to recover the graph structure. Recently, there is a surge of interest to learn algorithms directly based on data, and in this case, learn to map empirical covariance to the sparse precision matrix. However, it is a challenging task in this case, since the symmetric positive definiteness (SPD) and sparsity of the matrix are not easy to enforce in learned algorithms, and a direct mapping from data to precision matrix may contain many parameters. We propose a deep learning architecture, GLAD, which uses an Alternating Minimization (AM) algorithm as our model inductive bias, and learns the model parameters via supervised learning. We show that GLAD learns a very compact and effective model for recovering sparse graphs from data.

uGLAD: Sparse graph recovery by optimizing deep unrolled networks

Abstract: Probabilistic Graphical Models (PGMs) are generative models of complex systems. They rely on conditional independence assumptions between variables to learn sparse representations which can be visualized in a form of a graph. Such models are used for domain exploration and structure discovery in poorly understood domains. This work introduces a novel technique to perform sparse graph recovery by optimizing deep unrolled networks. Assuming that the input data $X\in\mathbb{R}^{M\times D}$ comes from an underlying multivariate Gaussian distribution, we apply a deep model on $X$ that outputs the precision matrix $\hat{\Theta}$, which can also be interpreted as the adjacency matrix. Our model, uGLAD, builds upon and extends the state-of-the-art model GLAD to the unsupervised setting. The key benefits of our model are (1) uGLAD automatically optimizes sparsity-related regularization parameters leading to better performance than existing algorithms. (2) We introduce multi-task learning based `consensus' strategy for robust handling of missing data in an unsupervised setting. We evaluate model results on synthetic Gaussian data, non-Gaussian data generated from Gene Regulatory Networks, and present a case study in anaerobic digestion.

Conditional Independence graph recovery methods with their implementation links. Additional information about their ability to recover multiple graphs (MG), handle large scale data (LS), handle missing values in data (MI) and to model time-series (TS) are mentioned alongside.

Table {{table:78e55477-32dd-49b8-a3b8-2d9d39813247}}  lists some of the prominent methods for CI graph recovery along with their recommended implementations. This compilation will help the readers choose the right models for their applications. Now that we have discussed some of the popular approaches to recover CI graphs, for the sake of completeness and wider adoption, we list out some of the potential applications that the CI graphs have been applied to in the past as well as hint at several unexplored opportunities to leverage them.


Simple and Deep Graph Convolutional Networks

Abstract: Graph convolutional networks (GCNs) are a powerful deep learning approach for graph-structured data. Recently, GCNs and subsequent variants have shown superior performance in various application areas on real-world datasets. Despite their success, most of the current GCN models are shallow, due to the {\em over-smoothing} problem. In this paper, we study the problem of designing and analyzing deep graph convolutional networks. We propose the GCNII, an extension of the vanilla GCN model with two simple yet effective techniques: {\em Initial residual} and {\em Identity mapping}. We provide theoretical and empirical evidence that the two techniques effectively relieves the problem of over-smoothing. Our experiments show that the deep GCNII model outperforms the state-of-the-art methods on various semi- and full-supervised tasks. Code is available at this https URL .

Representation Learning on Graphs with Jumping Knowledge Networks

Abstract: Recent deep learning approaches for representation learning on graphs follow a neighborhood aggregation procedure. We analyze some important properties of these models, and propose a strategy to overcome those. In particular, the range of "neighboring" nodes that a node's representation draws from strongly depends on the graph structure, analogous to the spread of a random walk. To adapt to local neighborhood properties and tasks, we explore an architecture -- jumping knowledge (JK) networks -- that flexibly leverages, for each node, different neighborhood ranges to enable better structure-aware representation. In a number of experiments on social, bioinformatics and citation networks, we demonstrate that our model achieves state-of-the-art performance. Furthermore, combining the JK framework with models like Graph Convolutional Networks, GraphSAGE and Graph Attention Networks consistently improves those models' performance.

Predict then Propagate: Graph Neural Networks meet Personalized PageRank

Abstract: Neural message passing algorithms for semi-supervised classification on graphs have recently achieved great success. However, for classifying a node these methods only consider nodes that are a few propagation steps away and the size of this utilized neighborhood is hard to extend. In this paper, we use the relationship between graph convolutional networks (GCN) and PageRank to derive an improved propagation scheme based on personalized PageRank. We utilize this propagation procedure to construct a simple model, personalized propagation of neural predictions (PPNP), and its fast approximation, APPNP. Our model's training time is on par or faster and its number of parameters on par or lower than previous models. It leverages a large, adjustable neighborhood for classification and can be easily combined with any neural network. We show that this model outperforms several recently proposed methods for semi-supervised classification in the most thorough study done so far for GCN-like models. Our implementation is available online.

Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective

Abstract: Graph convolutional networks (GCNs) have achieved promising performance on various graph-based tasks. However they suffer from over-smoothing when stacking more layers. In this paper, we present a quantitative study on this observation and develop novel insights towards the deeper GCN. First, we interpret the current graph convolutional operations from an optimization perspective and argue that over-smoothing is mainly caused by the naive first-order approximation of the solution to the optimization problem. Subsequently, we introduce two metrics to measure the over-smoothing on node-level tasks. Specifically, we calculate the fraction of the pairwise distance between connected and disconnected nodes to the overall distance respectively. Based on our theoretical and empirical analysis, we establish a universal theoretical framework of GCN from an optimization perspective and derive a novel convolutional kernel named GCN+ which has lower parameter amount while relieving the over-smoothing inherently. Extensive experiments on real-world datasets demonstrate the superior performance of GCN+ over state-of-the-art baseline methods on the node classification tasks.

Break the Ceiling: Stronger Multi-scale Deep Graph Convolutional Networks

Abstract: Recently, neural network based approaches have achieved significant improvement for solving large, complex, graph-structured problems. However, their bottlenecks still need to be addressed, and the advantages of multi-scale information and deep architectures have not been sufficiently exploited. In this paper, we theoretically analyze how existing Graph Convolutional Networks (GCNs) have limited expressive power due to the constraint of the activation functions and their architectures. We generalize spectral graph convolution and deep GCN in block Krylov subspace forms and devise two architectures, both with the potential to be scaled deeper but each making use of the multi-scale information in different ways. We further show that the equivalence of these two architectures can be established under certain conditions. On several node classification tasks, with or without the help of validation, the two new architectures achieve better performance compared to many state-of-the-art methods.

Towards Deeper Graph Neural Networks

Abstract: Graph neural networks have shown significant success in the field of graph representation learning. Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Nevertheless, one layer of these neighborhood aggregation methods only consider immediate neighbors, and the performance decreases when going deeper to enable larger receptive fields. Several recent studies attribute this performance deterioration to the over-smoothing issue, which states that repeated propagation makes node representations of different classes indistinguishable. In this work, we study this observation systematically and develop new insights towards deeper graph neural networks. First, we provide a systematical analysis on this issue and argue that the key factor compromising the performance significantly is the entanglement of representation transformation and propagation in current graph convolution operations. After decoupling these two operations, deeper graph neural networks can be used to learn graph node representations from larger receptive fields. We further provide a theoretical analysis of the above observation when building very deep models, which can serve as a rigorous and gentle description of the over-smoothing issue. Based on our theoretical and empirical analysis, we propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields. A set of experiments on citation, co-authorship, and co-purchase datasets have confirmed our analysis and insights and demonstrated the superiority of our proposed methods.

PairNorm: Tackling Oversmoothing in GNNs

Abstract: The performance of graph neural nets (GNNs) is known to gradually decrease with increasing number of layers. This decay is partly attributed to oversmoothing, where repeated graph convolutions eventually make node embeddings indistinguishable. We take a closer look at two different interpretations, aiming to quantify oversmoothing. Our main contribution is PairNorm, a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar. What is more, PairNorm is fast, easy to implement without any change to network architecture nor any additional parameters, and is broadly applicable to any GNN. Experiments on real-world graphs demonstrate that PairNorm makes deeper GCN, GAT, and SGC models more robust against oversmoothing, and significantly boosts performance for a new problem setting that benefits from deeper GNNs. Code is available at this https URL.

Scattering GCN: Overcoming Oversmoothness in Graph Convolutional Networks

Abstract: Graph convolutional networks (GCNs) have shown promising results in processing graph data by extracting structure-aware features. This gave rise to extensive work in geometric deep learning, focusing on designing network architectures that ensure neuron activations conform to regularity patterns within the input graph. However, in most cases the graph structure is only accounted for by considering the similarity of activations between adjacent nodes, which limits the capabilities of such methods to discriminate between nodes in a graph. Here, we propose to augment conventional GCNs with geometric scattering transforms and residual convolutions. The former enables band-pass filtering of graph signals, thus alleviating the so-called oversmoothing often encountered in GCNs, while the latter is introduced to clear the resulting features of high-frequency noise. We establish the advantages of the presented Scattering GCN with both theoretical results establishing the complementary benefits of scattering and GCN features, as well as experimental results showing the benefits of our method compared to leading graph neural networks for semi-supervised node classification, including the recently proposed GAT network that typically alleviates oversmoothing using graph attention mechanisms.

Towards Deeper Graph Neural Networks with Differentiable Group Normalization

Abstract: Graph neural networks (GNNs), which learn the representation of a node by aggregating its neighbors, have become an effective computational tool in downstream applications. Over-smoothing is one of the key issues which limit the performance of GNNs as the number of layers increases. It is because the stacked aggregators would make node representations converge to indistinguishable vectors. Several attempts have been made to tackle the issue by bringing linked node pairs close and unlinked pairs distinct. However, they often ignore the intrinsic community structures and would result in sub-optimal performance. The representations of nodes within the same community/class need be similar to facilitate the classification, while different classes are expected to be separated in embedding space. To bridge the gap, we introduce two over-smoothing metrics and a novel technique, i.e., differentiable group normalization (DGN). It normalizes nodes within the same group independently to increase their smoothness, and separates node distributions among different groups to significantly alleviate the over-smoothing issue. Experiments on real-world datasets demonstrate that DGN makes GNN models more robust to over-smoothing and achieves better performance with deeper GNNs.

Understanding and Resolving Performance Degradation in Deep Graph Convolutional Networks

Abstract: A Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation~(PROP) and a TRANsformation operation~(TRAN) for learning node representations over graph-structured data. Though powerful, GCNs tend to suffer performance drop when the model gets deep. Previous works focus on PROPs to study and mitigate this issue, but the role of TRANs is barely investigated. In this work, we study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works. We find that TRANs contribute significantly, or even more than PROPs, to declining performance, and moreover that they tend to amplify node-wise feature variance in GCNs, causing variance inflammation that we identify as a key factor for causing performance drop. Motivated by such observations, we propose a variance-controlling technique termed Node Normalization (NodeNorm), which scales each node's features using its own standard deviation. Experimental results validate the effectiveness of NodeNorm on addressing performance degradation of GCNs. Specifically, it enables deep GCNs to outperform shallow ones in cases where deep models are needed, and to achieve comparable results with shallow ones on 6 benchmark datasets. NodeNorm is a generic plug-in and can well generalize to other GNN architectures. Code is publicly available at https://github.com/miafei/NodeNorm.

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

Abstract: Over-fitting and over-smoothing are two main obstacles of developing deep Graph Convolutional Networks (GCNs) for node classification. In particular, over-fitting weakens the generalization ability on small dataset, while over-smoothing impedes model training by isolating output representations from the input features with the increase in network depth. This paper proposes DropEdge, a novel and flexible technique to alleviate both issues. At its core, DropEdge randomly removes a certain number of edges from the input graph at each training epoch, acting like a data augmenter and also a message passing reducer. Furthermore, we theoretically demonstrate that DropEdge either retards the convergence speed of over-smoothing or relieves the information loss caused by it. More importantly, our DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance. Extensive experiments on several benchmarks verify that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs. The effect of DropEdge on preventing over-smoothing is empirically visualized and validated as well. Codes will be made public upon the publication.

Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

Abstract: Graph convolutional networks (GCNs) have recently received wide attentions, due to their successful applications in different graph tasks and different domains. Training GCNs for a large graph, however, is still a challenge. Original full-batch GCN training requires calculating the representation of all the nodes in the graph per GCN layer, which brings in high computation and memory costs. To alleviate this issue, several sampling-based methods are proposed to train GCNs on a subset of nodes. Among them, the node-wise neighbor-sampling method recursively samples a fixed number of neighbor nodes, and thus its computation cost suffers from exponential growing neighbor size across layers; while the layer-wise importance-sampling method discards the neighbor-dependent constraints, and thus the nodes sampled across layer suffer from sparse connection problem. To deal with the above two problems, we propose a new effective sampling algorithm called LAyer-Dependent ImportancE Sampling (LADIES). Based on the sampled nodes in the upper layer, LADIES selects nodes that are in the neighborhood of these nodes and uses the constructed bipartite graph to compute the importance probability. Then, it samples a fixed number of nodes according to the probability for the whole layer, and recursively conducts such procedure per layer to construct the whole computation graph. We prove theoretically and experimentally, that our proposed sampling algorithm outperforms the previous sampling methods regarding both time and memory. Furthermore, LADIES is shown to have better generalization accuracy than original full-batch GCN, due to its stochastic nature.

Bayesian Graph Neural Networks with Adaptive Connection Sampling

Abstract: We propose a unified framework for adaptive connection sampling in graph neural networks (GNNs) that generalizes existing stochastic regularization methods for training GNNs. The proposed framework not only alleviates over-smoothing and over-fitting tendencies of deep GNNs, but also enables learning with uncertainty in graph analytic tasks with GNNs. Instead of using fixed sampling rates or hand-tuning them as model hyperparameters in existing stochastic regularization methods, our adaptive connection sampling can be trained jointly with GNN model parameters in both global and local fashions. GNN training with adaptive connection sampling is shown to be mathematically equivalent to an efficient approximation of training Bayesian GNNs. Experimental results with ablation studies on benchmark datasets validate that adaptively learning the sampling rate given graph training data is the key to boost the performance of GNNs in semi-supervised node classification, less prone to over-smoothing and over-fitting with more robust prediction.

Configurations of basic hyperparameters adopted to implement different approaches for training deep GNNs on Cora {{cite:bcb460c}}.Other graph datasets are refer to Section .

Aiming to establish such a fair benchmark, our first step is to thoroughly investigate the design philosophy and implementation details on dozens of poplular deep GNN training techniques, including various residual connections, graph normalization, and random dropping. The summarization could be founded in Tables {{table:6f8400d1-d594-4f02-88f9-40ca9100b320}} , {{table:2a41a82d-5a98-4b73-8d77-82f36359d0ed}} , {{table:0ea2e031-28aa-4471-bf6a-197f3d6fa2fc}} , and {{table:4acce5ed-493d-43b0-b729-89a55f16605e}} . Somehow unfortunately, we find that even sticking to the same dataset and GNN backbone, the hyperparameter configurations (e.g., hidden dimension, learning rate, weight decay, dropout rate, training epochs, early stopping patience) are highly inconsistently implemented, often varying case-to-case, which make it troubling to draw any fair conclusion.

Prerequisite: Unifying the Hyperparameter Configuration

We carefully examine previous implementations of deep GNNs {{cite:2363f96}}, {{cite:dee7a72}}, {{cite:0812184}}, {{cite:f7be3b7}}, {{cite:f2ad407}}, {{cite:a43ce6a}}, {{cite:cae2619}}, {{cite:82e037d}}, {{cite:35b159a}}, {{cite:b088268}}, and list all their basic hyperparameters in Table {{table:6f8400d1-d594-4f02-88f9-40ca9100b320}} , {{table:2a41a82d-5a98-4b73-8d77-82f36359d0ed}} , {{table:0ea2e031-28aa-4471-bf6a-197f3d6fa2fc}} , and {{table:4acce5ed-493d-43b0-b729-89a55f16605e}} . Those hyperparameters play a significant role in those methods' achievable performance, but their inconsistency challenges fair comparison of training techniques, which has been traditionally somehow overlooked in literature.


Variable Selection for Clustering and Classification

Abstract: As data sets continue to grow in size and complexity, effective and efficient techniques are needed to target important features in the variable space. Many of the variable selection techniques that are commonly used alongside clustering algorithms are based upon determining the best variable subspace according to model fitting in a stepwise manner. These techniques are often computationally intensive and can require extended periods of time to run; in fact, some are prohibitively computationally expensive for high-dimensional data. In this paper, a novel variable selection technique is introduced for use in clustering and classification analyses that is both intuitive and computationally efficient. We focus largely on applications in mixture model-based learning, but the technique could be adapted for use with various other clustering/classification methods. Our approach is illustrated on both simulated and real data, highlighted by contrasting its performance with that of other comparable variable selection techniques on the real data sets.

Variable selection in model-based clustering and discriminant analysis with a regularization approach

Abstract: Relevant methods of variable selection have been proposed in model-based clustering and classification. These methods are making use of backward or forward procedures to define the roles of the variables. Unfortunately, these stepwise procedures are terribly slow and make these variable selection algorithms inefficient to treat large data sets. In this paper, an alternative regularization approach of variable selection is proposed for model-based clustering and classification. In this approach, the variables are first ranked with a lasso-like procedure in order to avoid painfully slow stepwise algorithms. Thus, the variable selection methodology of Maugis et al (2009b) can be efficiently applied on high-dimensional data sets.

R packages for GMMs variable selection.

The available R packages for variable selection for Gaussian mixture models are: sparcl {{cite:ccd44a8}}, clustvarsel {{cite:a68a9f4}}, VarSelLCM {{cite:6a1b0f9}}, vscc {{cite:720af46}}, SelvarMix {{cite:6a0e3d4}}, and bclust {{cite:f43706c}}. Table {{table:631507d4-77d6-4f76-b55c-a7fd05aa54d4}}  lists them, with information about the method and the type of approach. It is also worth to mention the package ClustOfVar {{cite:6006d98}}. Rather than performing variable selection and obtain a classification of the data points, this package aims to find clusters of variables linked to a central synthetic variable obtained from a principal components decomposition of the data; for this reason, it will not be considered in the subsequent analysis. Lastly, we also point to the C++ softwares SelvarClusthttp://www.math.univ-toulouse.fr/~maugis/SelvarClustHomepage.html and SelvarClustIndephttp://www.math.univ-toulouse.fr/~maugis/SelvarClustIndepHomepage.html, which implement the modeling framework of {{cite:8ccf2bc}} and {{cite:97df12c}}, {{cite:d659974}}. We will not consider these software in the following analysis, since the aim of this paper is in reviewing R packages.

First we fit a Gaussian mixture model using the mclust package. The model selected is a 3-component mixture, with an ARI of 0.13 between the estimated classification and the historical period. Then we apply the variable selection methods listed in Table {{table:631507d4-77d6-4f76-b55c-a7fd05aa54d4}} . Figure {{figure:8f9c7255-048d-4241-8869-e17f426b93c8}}  displays the selected age groups for each method, alongside the chosen number of clusters and the ARI between the estimated classification and the historical period. Note that for sparcl the number of components needs to be set in advance and we fixed it to the number selected by mclust. Package bclust did not discard any variables, most likely due to the fact that adjacent age groups are highly correlated and the independence assumption of this method is unrealistic in this case. Also package vscc did not remove any of the variables. Package VarSelLCM retained the smallest subset of variables. However, the package obtained an ARI of zero and surprisingly selected only the oldest age intervals, those expected to have the least information, thus it seems to have detected a spurious solution. We ran the package multiple times and the other best solutions found were a model with 3 clusters and the same subset of variables, and a model with only one cluster and no variables discarded. The package seems to be sensitive to the initialization and, also in this case, this is an indication that the strong independence assumptions at the basis of the method implemented could be too restrictive. The other packages, sparcl, SelvarMix, and clustvarsel discarded age groups representing the old population. Among these, clustvarsel performed the most parsimonious selection, retaining only 11 age groups in the range from 0 to 49 years old, and with the highest ARI. The clusters are discriminated along the young and middle ages, as it would be expected since for elder ages the differences among death rates level off. Table {{table:d75ebd8d-1894-4233-bf11-7714b787ea9f}}  reports the cross-tabulation between the historical periods and the clustvarsel partition on the selected age intervals. The clusters have a meaningful interpretation in terms of the time dimension. Figure {{figure:09c81996-b582-4278-b510-16495a566ddd}}  shows the estimated cluster means on the age groups selected by clustvarsel. The mclust estimated cluster means are added for comparison. In particular, the mortality patterns of clusters 1, 3 and 4 estimated on the selected variables are close to the average patterns estimated on all the data. The extra cluster 2 seems to capture the mortality pattern related to the period corresponding to the two World wars. Moreover, cluster 5 has a mean rate similar to cluster 3, but higher death rate for age intervals above 20 years old and captures the extra-heterogeneity present in the more recent years.

Table {{table:6aaf8085-91c1-473e-ac12-fbd97ae952e3}}  reports the computing time of the packages considered in Table {{table:631507d4-77d6-4f76-b55c-a7fd05aa54d4}}  on a Dell machine with Intel Core i7-3770 CPU @3.40GHz{{formula:48f5a9b1-2fb5-42c9-8cbf-d4128e0c1c1c}} 8. The packages are listed from the fastest to the slowest. vscc is the fastest among the packages considered, but it did not discard any of the age intervals. Package sparcl is the second fastest. In the case of highly correlated variables, clustvarsel performs the most parsimonious selection and obtains the best classification, with an acceptable computing time. Note, however, that in using clustvarsel we considered only 4 out of the 14 covariance models available, as some of them may require a long computational time to be fitted and to consider a set of models consistent with SelvarMix. Therefore, aside from vscc, sparcl is the fastest and as such it is particularly suitable to perform variable selection in high-dimensional settings where it is reasonable to assume local independence among the variables. Nevertheless, the package requires the number of clusters to be specified in advance, while clustvarsel and SelvarMix automatically select such number. bclust and VarSelLCM are the slowest in this example, and care may need to be taken when using these packages on highly correlated data, since they make use of independence assumptions that could be too restrictive. Lastly, we point that clustvarsel, SelvarMix and VarSelLCM allow for parallelization of the computations.


Disentangling Monocular 3D Object Detection

Abstract: In this paper we propose an approach for monocular 3D object detection from a single RGB image, which leverages a novel disentangling transformation for 2D and 3D detection losses and a novel, self-supervised confidence score for 3D bounding boxes. Our proposed loss disentanglement has the twofold advantage of simplifying the training dynamics in the presence of losses with complex interactions of parameters, and sidestepping the issue of balancing independent regression terms. Our solution overcomes these issues by isolating the contribution made by groups of parameters to a given loss, without changing its nature. We further apply loss disentanglement to another novel, signed Intersection-over-Union criterion-driven loss for improving 2D detection results. Besides our methodological innovations, we critically review the AP metric used in KITTI3D, which emerged as the most important dataset for comparing 3D detection results. We identify and resolve a flaw in the 11-point interpolated AP metric, affecting all previously published detection results and particularly biases the results of monocular 3D detection. We provide extensive experimental evaluations and ablation studies and set a new state-of-the-art on the KITTI3D Car class.

Probabilistic and Geometric Depth: Detecting Objects in Perspective

Abstract: 3D object detection is an important capability needed in various practical applications such as driver assistance systems. Monocular 3D detection, as a representative general setting among image-based approaches, provides a more economical solution than conventional settings relying on LiDARs but still yields unsatisfactory results. This paper first presents a systematic study on this problem. We observe that the current monocular 3D detection can be simplified as an instance depth estimation problem: The inaccurate instance depth blocks all the other 3D attribute predictions from improving the overall detection performance. Moreover, recent methods directly estimate the depth based on isolated instances or pixels while ignoring the geometric relations across different objects. To this end, we construct geometric relation graphs across predicted objects and use the graph to facilitate depth estimation. As the preliminary depth estimation of each instance is usually inaccurate in this ill-posed setting, we incorporate a probabilistic representation to capture the uncertainty. It provides an important indicator to identify confident predictions and further guide the depth propagation. Despite the simplicity of the basic idea, our method, PGD, obtains significant improvements on KITTI and nuScenes benchmarks, achieving 1st place out of all monocular vision-only methods while still maintaining real-time efficiency. Code and models will be released at https://github.com/open-mmlab/mmdetection3d.

Is Pseudo-Lidar needed for Monocular 3D Object detection?

Abstract: Recent progress in 3D object detection from single images leverages monocular depth estimation as a way to produce 3D pointclouds, turning cameras into pseudo-lidar sensors. These two-stage detectors improve with the accuracy of the intermediate depth estimation network, which can itself be improved without manual labels via large-scale self-supervised learning. However, they tend to suffer from overfitting more than end-to-end methods, are more complex, and the gap with similar lidar-based detectors remains significant. In this work, we propose an end-to-end, single stage, monocular 3D object detector, DD3D, that can benefit from depth pre-training like pseudo-lidar methods, but without their limitations. Our architecture is designed for effective information transfer between depth estimation and 3D detection, allowing us to scale with the amount of unlabeled pre-training data. Our method achieves state-of-the-art results on two challenging benchmarks, with 16.34% and 9.28% AP for Cars and Pedestrians (respectively) on the KITTI-3D benchmark, and 41.5% mAP on NuScenes.

Perfromance of category level monocular 3D object detection methods on Nuscenes dataset.

Category-Level Monocular 3D Object Detection

Till now, the recent state-of-the-art monocular 3D object detection methods have been introduced. Table {{table:d4d1e4fb-da91-4e32-a741-6bbfe78e6307}}  presents an overview and a complete comparison of representative works. The results are about the performance of these methods on KITTI val1, val2, and test split. The mAP@IoU75 of Car class, which is the most widely concerned category in monocular 3D object detection, is reported. The results about val1 split and val2 split are reported using the {{formula:f5f9a59a-18bf-4623-8aa3-24b6df253030}}  metric to provide a complete comparison for all published related works. The results about the test split are reported using the {{formula:a6bdca2a-b896-4e83-9074-6cc8ef3eaa85}}  metric, which is the latest standard evaluation metric of KITTI's official leaderboard. Table {{table:524bd5e2-61bb-46fa-ab8e-149488b04ab3}}  shows a comparison between current state-of-the-art methods on Nuscenes datasets. The metrics are mAP and NDS.


Sunnybrook Cardiac Data {{cite:425edb9}}

Fast Predictive Image Registration

Abstract: We present a method to predict image deformations based on patch-wise image appearance. Specifically, we design a patch-based deep encoder-decoder network which learns the pixel/voxel-wise mapping between image appearance and registration parameters. Our approach can predict general deformation parameterizations, however, we focus on the large deformation diffeomorphic metric mapping (LDDMM) registration model. By predicting the LDDMM momentum-parameterization we retain the desirable theoretical properties of LDDMM, while reducing computation time by orders of magnitude: combined with patch pruning, we achieve a 1500x/66x speed up compared to GPU-based optimization for 2D/3D image registration. Our approach has better prediction accuracy than predicting deformation or velocity fields and results in diffeomorphic transformations. Additionally, we create a Bayesian probabilistic version of our network, which allows evaluation of deformation field uncertainty through Monte Carlo sampling using dropout at test time. We show that deformation uncertainty highlights areas of ambiguous deformations. We test our method on the OASIS brain image dataset in 2D and 3D.

Medical Image Synthesis with Context-Aware Generative Adversarial Networks

Abstract: Computed tomography (CT) is critical for various clinical applications, e.g., radiotherapy treatment planning and also PET attenuation correction. However, CT exposes radiation during acquisition, which may cause side effects to patients. Compared to CT, magnetic resonance imaging (MRI) is much safer and does not involve any radiations. Therefore, recently, researchers are greatly motivated to estimate CT image from its corresponding MR image of the same subject for the case of radiotherapy planning. In this paper, we propose a data-driven approach to address this challenging problem. Specifically, we train a fully convolutional network to generate CT given an MR image. To better model the nonlinear relationship from MRI to CT and to produce more realistic images, we propose to use the adversarial training strategy and an image gradient difference loss function. We further apply AutoContext Model to implement a context-aware generative adversarial network. Experimental results show that our method is accurate and robust for predicting CT images from MRI images, and also outperforms three state-of-the-art methods under comparison.

A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstruction

Abstract: Due to the potential risk of inducing cancers, radiation dose of X-ray CT should be reduced for routine patient scanning. However, in low-dose X-ray CT, severe artifacts usually occur due to photon starvation, beamhardening, etc, which decrease the reliability of diagnosis. Thus, high quality reconstruction from low-dose X-ray CT data has become one of the important research topics in CT community. Conventional model-based denoising approaches are, however, computationally very expensive, and image domain denoising approaches hardly deal with CT specific noise patterns. To address these issues, we propose an algorithm using a deep convolutional neural network (CNN), which is applied to wavelet transform coefficients of low-dose CT images. Specifically, by using a directional wavelet transform for extracting directional component of artifacts and exploiting the intra- and inter-band correlations, our deep network can effectively suppress CT specific noises. Moreover, our CNN is designed to have various types of residual learning architecture for faster network training and better denoising. Experimental results confirm that the proposed algorithm effectively removes complex noise patterns of CT images, originated from the reduced X-ray dose. In addition, we show that wavelet domain CNN is efficient in removing the noises from low-dose CT compared to an image domain CNN. Our results were rigorously evaluated by several radiologists and won the second place award in 2016 AAPM Low-Dose CT Grand Challenge. To the best of our knowledge, this work is the first deep learning architecture for low-dose CT reconstruction that has been rigorously evaluated and proven for its efficacy.

End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network

Abstract: In this work we propose a deep learning network for deformable image registration (DIRNet). The DIRNet consists of a convolutional neural network (ConvNet) regressor, a spatial transformer, and a resampler. The ConvNet analyzes a pair of fixed and moving images and outputs parameters for the spatial transformer, which generates the displacement vector field that enables the resampler to warp the moving image to the fixed image. The DIRNet is trained end-to-end by unsupervised optimization of a similarity metric between input image pairs. A trained DIRNet can be applied to perform registration on unseen image pairs in one pass, thus non-iteratively. Evaluation was performed with registration of images of handwritten digits (MNIST) and cardiac cine MR scans (Sunnybrook Cardiac Data). The results demonstrate that registration with DIRNet is as accurate as a conventional deformable image registration method with substantially shorter execution times.

Summary of influential papers appeared in 2016 and 2017 (based on Google Scholar citation index in January 2019) that exploit deep learning for the Registration task in Medical Image Analysis.

Yan  et al. {{cite:b8bad43}} presented an Adversarial Image Registration (AIR) method for multi-modal image MR-TRUS registration {{cite:aa20fcd}}. They trained two deep networks concurrently, one for the generator component of the adversarial framework, and the other for the discriminator component. In their work, the authors learned not only an image registration network but also a so-called metric network which computes the quality of image registration.
The data used in their experimentation consists of 763 sets of 3D TRUS volume and 2D MR volume with 512x512x26 voxels. The developed AIR network is also evaluated on clinical datasets acquired through image-fusion guided prostate biopsy procedures.
For the visualization of 3D medical image data Zhao et al. {{cite:9ecdc01}} recently proposed a deep learning based technique, named Respond-weighted Class Activation Mapping (Respond-CAM). As compared to Grade-CAM {{cite:9f99691}} they claim better performance.
Elss et al. {{cite:c675f21}} also employed Convolutional networks for single phase image motion in cardiac CT 2D/3D images. They trained regression network to successfully learn 2D motion estimation vectors. We also summarize few worth noting contributions from the years 2016 and 2017 in Table {{table:67c71e97-428e-4daa-ada9-4b2b039e05f9}} .


Disfluency Detection Using a Bidirectional LSTM

Abstract: We introduce a new approach for disfluency detection using a Bidirectional Long-Short Term Memory neural network (BLSTM). In addition to the word sequence, the model takes as input pattern match features that were developed to reduce sensitivity to vocabulary size in training, which lead to improved performance over the word sequence alone. The BLSTM takes advantage of explicit repair states in addition to the standard reparandum states. The final output leverages integer linear programming to incorporate constraints of disfluency structure. In experiments on the Switchboard corpus, the model achieves state-of-the-art performance for both the standard disfluency detection task and the correction detection task. Analysis shows that the model has better detection of non-repetition disfluencies, which tend to be much harder to detect.

Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection

Abstract: With the increased applications of automatic speech recognition (ASR) in recent years, it is essential to automatically insert punctuation marks and remove disfluencies in transcripts, to improve the readability of the transcripts as well as the performance of subsequent applications, such as machine translation, dialogue systems, and so forth. In this paper, we propose a Controllable Time-delay Transformer (CT-Transformer) model that jointly completes the punctuation prediction and disfluency detection tasks in real time. The CT-Transformer model facilitates freezing partial outputs with controllable time delay to fulfill the real-time constraints in partial decoding required by subsequent applications. We further propose a fast decoding strategy to minimize latency while maintaining competitive performance. Experimental results on the IWSLT2011 benchmark dataset and an in-house Chinese annotated dataset demonstrate that the proposed approach outperforms the previous state-of-the-art models on F-scores and achieves a competitive inference speed.

Summary of previous stutter disfluency classification methods.

Table {{table:67dd31ad-789a-41f2-9355-6d469b505a6f}}  provides a summary of the related works on disfluency detection and classification. It can be observed and concluded that disfluency classification has been progressing in one of two fronts i) end-to-end speech-based methods, or ii) language-based models relying on an ASR pre-processing step. Our work in this paper is positioned in the first category in order to avoid the reliance on an ASR step. Moreover, from Table {{table:67dd31ad-789a-41f2-9355-6d469b505a6f}} , we observe that although progresses is being made in the area of speech disfluency recognition, the lack of available data remains a hindrance to potential further achievements in the field.


Character-level Convolutional Networks for Text Classification

Abstract: This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification. We constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results. Comparisons are offered against traditional models such as bag of words, n-grams and their TFIDF variants, and deep learning models such as word-based ConvNets and recurrent neural networks.

MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages

Abstract: We present the MASSIVE dataset–Multilingual Amazon Slu resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation. MASSIVE contains 1M realistic, parallel, labeled virtual assistant utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE was created by tasking professional translators to localize the English-only SLURP dataset into 50 typologically diverse languages from 29 genera. We also present modeling results on XLM-R and mT5, including exact match accuracy, intent classification accuracy, and slot-filling F1 score. We have released our dataset, modeling code, and models publicly.

List of datasets used for classification evaluation.

In order to perform a comprehensive evaluation, we evaluate classification performance on four different datasets with drastically different dataset sizes, number of languages, classification tasks, and text lengths, as summarized in Table {{table:410fcc66-d8f0-4fba-8af4-c58fa7300f79}} . For example, the MASSIVE {{cite:85ba21f}} intent classification dataset has very short sentences but a lot of languages (51), whereas the Yelp Reviews dataset has significantly longer texts but is monolingual.


Coordinates where asset increase or decrease

No Silk Road for Online Gamers!: Using Social Network Analysis to Unveil Black Markets in Online Games

Abstract: Online game involves a very large number of users who are interconnected and interact with each other via the Internet. We studied the characteristics of exchanging virtual goods with real money through processes called "real money trading (RMT)." This exchange might influence online game user behaviors and cause damage to the reputation of game companies. We examined in-game transactions to reveal RMT by constructing a social graph of virtual goods exchanges in an online game and identifying network communities of users. We analyzed approximately 6,000,000 transactions in a popular online game and inferred RMT transactions by comparing the RMT transactions crawled from an out-game market. Our findings are summarized as follows: (1) the size of the RMT market could be approximately estimated; (2) professional RMT providers typically form a specific network structure (either star-shape or chain) in the trading network, which can be used as a clue for tracing RMT transactions; and (3) the observed RMT market has evolved over time into a monopolized market with a small number of large-sized virtual goods providers.

Show Me Your Account: Detecting MMORPG Game Bot Leveraging Financial Analysis with LSTM

Abstract: With the rapid growth of MMORPG market, game bot detection has become an essential task for maintaining stable in-game ecosystem. To classify bots from normal users, detection methods are proposed in both game client and server-side. Among various classification methods, data mining method in server-side captured unique characteristics of bots efficiently. For features used in data mining, behavioral and social actions of character are analyzed with numerous algorithms. However, bot developers can evade the previous detection methods by changing bot's activities continuously. Eventually, overall maintenance cost increases because the selected features need to be updated along with the change of bot's behavior. To overcome this limitation, we propose improved bot detection method with financial analysis. As bot's activity absolutely necessitates the change of financial status, analyzing financial fluctuation effectively captures bots as a key feature. We trained and tested model with actual data of Aion, a leading MMORPG in Asia. Leveraging that LSTM efficiently recognizes time-series movement of data, we achieved meaningful detection performance. Further on this model, we expect sustainable bot detection system in the near future.

Researches on server-side bot detection method

Among previous works, server-side method analyzes game play data to capture unique pattern of bots. We categorized features used in bot detection as Table {{table:87fe64a0-857c-4e7e-ba88-591514f42fe4}} , by dividing into two streams: sufficient condition and necessary condition. The sufficient conditions are set of features that bot's activity probably creates, but not absolutely accompanies. As the sufficient conditions include distinctive pattern of bots, we can identify bots during certain period of time. However, analysis of the sufficient conditions require repetitive updates as these conditions are not consistent. If bot developers modify bot's behavior, model also necessitates update to capture changed pattern. On the other hand, the necessary conditions are features must happen as a consequence of bot's behavior. As bots are designed to accumulate wealth efficiently, specific actions such as transaction among bots absolutely happen. Although bot developers change bot's behavior, pattern of wealth accumulation itself still exists. Thus, we can detect bots leveraging necessary conditions in consistent way.


DialBERT: A Hierarchical Pre-Trained Model for Conversation Disentanglement

Abstract: Disentanglement is a problem in which multiple conversations occur in the same channel simultaneously, and the listener should decide which utterance is part of the conversation he will respond to. We propose a new model, named Dialogue BERT (DialBERT), which integrates local and global semantics in a single stream of messages to disentangle the conversations that mixed together. We employ BERT to capture the matching information in each utterance pair at the utterance-level, and use a BiLSTM to aggregate and incorporate the context-level information. With only a 3% increase in parameters, a 12% improvement has been attained in comparison to BERT, based on the F1-Score. The model achieves a state-of-the-art result on the a new dataset proposed by IBM and surpasses previous work by a substantial margin.

Online Conversation Disentanglement with Pointer Networks

Abstract: Huge amounts of textual conversations occur online every day, where multiple conversations take place concurrently. Interleaved conversations lead to difficulties in not only following the ongoing discussions but also extracting relevant information from simultaneous messages. Conversation disentanglement aims to separate intermingled messages into detached conversations. However, existing disentanglement methods rely mostly on handcrafted features that are dataset specific, which hinders generalization and adaptability. In this work, we propose an end-to-end online framework for conversation disentanglement that avoids time-consuming domain-specific feature engineering. We design a novel way to embed the whole utterance that comprises timestamp, speaker, and message text, and proposes a custom attention mechanism that models disentanglement as a pointing problem while effectively capturing inter-utterance interactions in an end-to-end fashion. We also introduce a joint-learning objective to better capture contextual information. Our experiments on the Ubuntu IRC dataset show that our method achieves state-of-the-art performance in both link and conversation prediction tasks.

Candidate disentanglement models.

By searching through the literature published in the representative venues (Computational Linguistics, NAACL, ACL, IJCAI, EMNLP and SIGIR) for the last 15 years, we choose eight approaches as the candidate models for our study. Their code accessibility, dataset accessibility, and learning technologies are summarized in Table {{table:8f8fb1cd-c863-4418-9f45-b1ad25fb91f5}} . From the table, we can observe that five out of the eight models provide public assess to their source code and dataset. Moreover, these five models also utilize more advanced technologies such as deep neural network. Thus, we select FF, BiLSTM, BERT, E2E and PtrNet as the state-of-the-art (SOTA) models to be compared in our study. Each model is described in detail as follows.


Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Learning Visual N-Grams from Web Data

Abstract: Real-world image recognition systems need to recognize tens of thousands of classes that constitute a plethora of visual concepts. The traditional approach of annotating thousands of images per class for training is infeasible in such a scenario, prompting the use of webly supervised data. This paper explores the training of image-recognition systems on large numbers of images and associated user comments, without using manually labeled images. In particular, we develop visual n-gram models that can predict arbitrary phrases that are relevant to the content of an image. Our visual n-gram models are feed-forward convolutional networks trained using new loss functions that are inspired by n-gram models commonly used in language modeling. We demonstrate the merits of our models in phrase prediction, phrase-based image retrieval, relating images and captions, and zero-shot transfer.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we address the challenge of land use and land cover classification using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible provided in the Earth observation program Copernicus. We present a novel dataset based on Sentinel-2 satellite images covering 13 spectral bands and consisting out of 10 classes with in total 27,000 labeled and geo-referenced images. We provide benchmarks for this novel dataset with its spectral bands using state-of-the-art deep Convolutional Neural Network (CNNs). With the proposed novel dataset, we achieved an overall classification accuracy of 98.57%. The resulting classification system opens a gate towards a number of Earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

Datasets in our experiments.

The details of the 12 downstream datasets are shown in Tabel {{table:924c6bbd-d7e5-4e41-8dde-a51fcdd2b2c3}} . The accuracy metric of each dataset follows CLIP {{cite:5e0223c}}.


Determination of customer buy and leave decision factors

Fault analysis in steel plates manufacturing

Why to Buy Insurance? An Explainable Artificial Intelligence Approach

Abstract: We propose an Explainable AI model that can be employed in order to explain why a customer buys or abandons a non-life insurance coverage. The method consists in applying similarity clustering to the Shapley values that were obtained from a highly accurate XGBoost predictive classification algorithm. Our proposed method can be embedded into a technologically-based insurance service (Insurtech), allowing to understand, in real time, the factors that most contribute to customers’ decisions, thereby gaining proactive insights on their needs. We prove the validity of our model with an empirical analysis that was conducted on data regarding purchases of insurance micro-policies. Two aspects are investigated: the propensity to buy an insurance policy and the risk of churn of an existing customer. The results from the analysis reveal that customers can be effectively and quickly grouped according to a similar set of characteristics, which can predict their buying or churn behaviour well.

Explainable Artificial Intelligence Based Fault Diagnosis and Insight Harvesting for Steel Plates Manufacturing

Abstract: With the advent of Industry 4.0, Data Science and Explainable Artificial Intelligence (XAI) has received considerable intrest in recent literature. However, the entry threshold into XAI, in terms of computer coding and the requisite mathematical apparatus, is really high. For fault diagnosis of steel plates, this work reports on a methodology of incorporating XAI based insights into the Data Science process of development of high precision classifier. Using Synthetic Minority Oversampling Technique (SMOTE) and notion of medoids, insights from XAI tools viz. Ceteris Peribus profiles, Partial Dependence and Breakdown profiles have been harvested. Additionally, insights in the form of IF-THEN rules have also been extracted from an optimized Random Forest and Association Rule Mining. Incorporating all the insights into a single ensemble classifier, a 10 fold cross validated performance of 94% has been achieved. In sum total, this work makes three main contributions viz.: methodology based upon utilization of medoids and SMOTE, of gleaning insights and incorporating into model development process. Secondly the insights themselves are contribution, as they benefit the human experts of steel manufacturing industry, and thirdly a high precision fault diagnosis classifier has been developed.

Overview of XAI-based methods in Industry 4.0 adapted from {{cite:3adc9ba}}

With the increasing availability of data and compute resources in the age of Industry 4.0, AI applications in the industry become more prevalent. Due to the value at stake in industrial AI applications, XAI not only becomes increasingly important to ensure buy-in from stakeholders but also to understand the driving factors of outcomes in complex processes [{{cite:ad99fc8}} {{cite:3adc9ba}}]. {{cite:3adc9ba}} conducted a survey on AI and XAI applications in Industry 4.0. Table {{table:504b1c2b-ad0e-48cb-b836-1828253d0481}}  summarises the publications surveyed by {{cite:3adc9ba}} that were classified in the XAI area. Notably, the main XAI method used is SHAP {{cite:cd923a1}} in four of seven papers with the shared objective of determining the features that have a significant influence on the outcome of the model. We, therefore, use SHAP as the benchmark in the following chapters of this paper. The remaining three papers use Local Interpretable Model Agnostic Explanation (LIME) {{cite:6546f2a}} and bespoke XAI methods. The use cases vary from detecting the causes of quality variance to understanding why customers decided to buy a product but still share the same common question of "Who or what contributes to the (un)desired outcome?".

As highlighted in Table {{table:504b1c2b-ad0e-48cb-b836-1828253d0481}} , SHAP values are one of the most popular tools for XAI in the industry due to their performance and wide availability for different models ranging from trees to deep neural networks. Based on cooperative game theory, SHAP calculates the contribution of each feature to the model prediction {{cite:cd923a1}}. Insights on feature contribution can be used in the industry to detect the sources of quality variance as performed by {{cite:463fcdc}}, who used SHAP values to detect the processes to prioritise for quality improvement and were able to reduce yield loss by 21.7%.


Wrapped Subset Evaluation, Feature selection (12)

Improving the Accuracy of Predicting Bank Depositor’s Behavior Using a Decision Tree

Abstract: Telemarketing is a widely adopted direct marketing technique in banks. Since customers hardly respond positively, data prediction models can help in selecting the most likely prospective customers. We aim to develop a classifier accuracy to predict which customer will subscribe to a long-term deposit proposed by a bank. Accordingly, this paper focuses on a combination of resampling, in order to reduce the imbalanced data, using feature selection, to reduce the complexity of data computing and dimension reduction of inefficiency data modeling. The performed operation has shown an improvement in the performance of the classification algorithm in terms of accuracy. The experimental results were run on a real bank dataset and the J48 decision tree achieved 94.39% accuracy prediction, with 0.975 sensitivity and 0.709 specificity, showing better results when compared to other approaches reported in the existing literature, such as logistic regression (91.79 accuracy; 0.975 sensitivity; 0.495 specificity) and Naive Bayes classifier (90.82% accuracy; 0.961 sensitivity; 0.507 specificity). Furthermore, our resampling and feature selection approach resulted in improved accuracy (94.39%) when compared to a state-of-the-art approach based on a fuzzy algorithm (92.89%).

Business Analytics in Telemarketing: Cost-Sensitive Analysis of Bank Campaigns Using Artificial Neural Networks

Abstract: The banking industry has been seeking novel ways to leverage database marketing efficiency. However, the nature of bank marketing data hindered the researchers in the process of finding a reliable analytical scheme. Various studies have attempted to improve the performance of Artificial Neural Networks in predicting clients’ intentions but did not resolve the issue of imbalanced data. This research aims at improving the performance of predicting the willingness of bank clients to apply for a term deposit in highly imbalanced datasets. It proposes enhanced Artificial Neural Network models (i.e., cost-sensitive) to mitigate the dramatic effects of highly imbalanced data, without distorting the original data samples. The generated models are evaluated, validated, and consequently compared to different machine-learning models. A real-world telemarketing dataset from a Portuguese bank is used in all the experiments. The best prediction model achieved 79% of geometric mean, and misclassification errors were minimized to 0.192, 0.229 of Type I & Type II Errors, respectively. In summary, an interesting Meta-Cost method improved the performance of the prediction model without imposing significant processing overhead or altering original data samples.

Summary of bank telemarketing related works.

Moro et al. made available a rich real-world dataset containing telemarketing campaign data from a Portuguese bank {{cite:1b74ced}}, which they enriched later with further national socio-economic metrics {{cite:8007387}}. Since the release of the initial dataset various researchers have been attempting to generate prediction models and descriptive analytics; to assess the marketing decision-making process. However, due to the imbalance distribution of the data and other characteristics, non of the related works was able to provide solutions that handle all the challenges faced in modeling this business issue {{cite:1b74ced}}, {{cite:23f3388}}, {{cite:8007387}}. Interestingly, most of the researchers agree that not all the dataset features have the same importance in understanding the intentions of the clients {{cite:1b74ced}}, {{cite:8007387}}, {{cite:f0b6954}}, {{cite:5c5b2da}}, {{cite:f8bf065}}, {{cite:bbbaec5}}. Table {{table:393748c4-6d04-42ad-bf46-57ab4b4dc253}}  describes, in brief, some of the related works and the best-attained prediction performance in terms of Geometric Mean (GMean) and Type I Error. Some of the works listed in Table {{table:393748c4-6d04-42ad-bf46-57ab4b4dc253}}  reported the modeling performance of different approaches and Machine Learning (ML) algorithms {{cite:a3e7032}}, {{cite:f0b6954}}, {{cite:046405c}} and some attempted to understand the significance of the specific dataset features in predicting the willingness of the client to accept an offer {{cite:8007387}}, {{cite:5c5b2da}}, {{cite:f8bf065}}.

As shown in Table {{table:393748c4-6d04-42ad-bf46-57ab4b4dc253}} , recent studies of bank telemarketing, approach the problem of predicting clients' willingness to enroll in term deposits from two angles, Feature selection, and imbalanced data handling. Feature selection reduces the complexity of a prediction model. Proper handling of imbalanced data allows for better generalization of the prediction model to handle real-world scenarios without exhibiting any bias towards any of the target classes. Xie et al. {{cite:f0b6954}} proposed normalization to select features most effectively influencing a client's response to a telemarketing offer made by a bank. Further, they utilized “SMOTE”, oversampling, to extend the minority class to overcome issues related to data imbalances. Besides losing the generality of the prediction model due to the oversampling, we argue that our approach outperforms {{cite:f0b6954}} by digging deep to identify feature-level characteristics that help in predicting client intent. Moro, Cortez, and Rita proposed a feature selection approach based on sensitivity analysis and expert analysis {{cite:8007387}}. The need for human expertise remains a raised question. Compared to the literature and our proposed solution, {{cite:8007387}} make modest progress in terms of type I error. The only justification is overlooking the imbalanced nature of the data. Although features selection improved model accuracy (i.e., GMean values for the last two rows in Table {{table:393748c4-6d04-42ad-bf46-57ab4b4dc253}} ), this improvement was at the expense of missing clients interested in long-term deposit offers.


Architectural Tricks for Deep Learning in Remote Photoplethysmography

Abstract: Architectural improvements are studied for convolutional network performing estimation of heart rate (HR) values on color signal patches. Color signals are time series of color components averaged over facial regions recorded by webcams in two scenarios: Stationary (without motion of a person) and Mixed Motion (different motion patterns of a person). HR estimation problem is addressed as a classification task, where classes correspond to different heart rate values within the admissible range of [40; 125] bpm. Both adding convolutional filtering layers after fully connected layers and involving combined loss function where first component is a cross entropy and second is a squared error between the network output and smoothed one-hot vector, lead to better performance of HR estimation model in Stationary and Mixed Motion scenarios.

EVM-CNN: Real-Time Contactless Heart Rate Estimation From Facial Video

Abstract: With the increase in health consciousness, noninvasive body monitoring has aroused interest among researchers. As one of the most important pieces of physiological information, researchers have remotely estimated the heart rate (HR) from facial videos in recent years. Although progress has been made over the past few years, there are still some limitations, like the processing time increasing with accuracy and the lack of comprehensive and challenging datasets for use and comparison. Recently, it was shown that HR information can be extracted from facial videos by spatial decomposition and temporal filtering. Inspired by this, a new framework is introduced in this paper to remotely estimate the HR under realistic conditions by combining spatial and temporal filtering and a convolutional neural network. Our proposed approach shows better performance compared with the benchmark on the MMSE-HR dataset in terms of both the average HR estimation and short-time HR estimation. High consistency in short-time HR estimation is observed between our method and the ground truth.

List of retrieved articles.

Table {{table:84f41ebd-c662-4d57-bf1a-d6a3d213ecb7}}  lists the twenty articles that were selected. These articles were reviewed in order to answer the research question Q2, and our findings are provided in the subsections that follow: