Indonesian Wikipedia, News and Web corpus

Finnish News, Online discussion and Common Crawl Corpus

Thai Wikipedia , Social media posts, Books and reviews corpus

Korean Wikipedia, News, Internet crawl and Books corpus

IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP

Abstract: Although the Indonesian language is spoken by almost 200 million people and the 10th most spoken language in the world, it is under-represented in NLP research. Previous work on Indonesian has been hampered by a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In this work, we release the IndoLEM dataset comprising seven tasks for the Indonesian language, spanning morpho-syntax, semantics, and discourse. We additionally release IndoBERT, a new pre-trained language model for Indonesian, and evaluate it over IndoLEM, in addition to benchmarking it against existing resources. Our experiments show that IndoBERT achieves state-of-the-art performance over most of the tasks in IndoLEM.

IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

Abstract: Although Indonesian is known to be the fourth most frequently used language over the internet, the research progress on this language in the natural language processing (NLP) is slow-moving due to a lack of available resources. In response, we introduce the first-ever vast resource for the training, evaluating, and benchmarking on Indonesian natural language understanding (IndoNLU) tasks. IndoNLU includes twelve tasks, ranging from single sentence classification to pair-sentences sequence labeling with different levels of complexity. The datasets for the tasks lie in different domains and styles to ensure task diversity. We also provide a set of Indonesian pre-trained models (IndoBERT) trained from a large and clean Indonesian dataset Indo4B collected from publicly available sources such as social media texts, blogs, news, and websites. We release baseline models for all twelve tasks, as well as the framework for benchmark evaluation, and thus it enables everyone to benchmark their system performances.

PhoBERT: Pre-trained language models for Vietnamese

Abstract: We present PhoBERT with two versions, PhoBERT-base and PhoBERT-large, the first public large-scale monolingual language models pre-trained for Vietnamese. Experimental results show that PhoBERT consistently outperforms the recent best pre-trained multilingual model XLM-R (Conneau et al., 2020) and improves the state-of-the-art in multiple Vietnamese-specific NLP tasks including Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference. We release PhoBERT to facilitate future research and downstream applications for Vietnamese NLP. Our PhoBERT models are available at https://github.com/VinAIResearch/PhoBERT

RobBERT: a Dutch RoBERTa-based Language Model

Abstract: Pre-trained language models have been dominating the field of natural language processing in recent years, and have led to significant performance gains for various complex natural language tasks. One of the most prominent pre-trained language models is BERT, which was released as an English as well as a multilingual version. Although multilingual BERT performs well on many tasks, recent studies show that BERT models trained on a single language significantly outperform the multilingual version. Training a Dutch BERT model thus has a lot of potential for a wide range of Dutch NLP tasks. While previous approaches have used earlier implementations of BERT to train a Dutch version of BERT, we used RoBERTa, a robustly optimized BERT approach, to train a Dutch language model called RobBERT. We measured its performance on various tasks as well as the importance of the fine-tuning dataset size. We also evaluated the importance of language-specific tokenizers and the model’s fairness. We found that RobBERT improves state-of-the-art results for various tasks, and especially significantly outperforms other models when dealing with smaller datasets. These results indicate that it is a powerful pre-trained model for a large variety of Dutch language tasks. The pre-trained and fine-tuned models are publicly available to support further downstream Dutch NLP applications.

The birth of Romanian BERT

Abstract: Large-scale pretrained language models have become ubiquitous in Natural Language Processing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus composition and cleaning, the model training process, as well as an extensive evaluation of the model on various Romanian datasets. We open source not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process.

FlauBERT: Unsupervised Language Model Pre-training for French

Abstract: Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research community for further reproducible experiments in French NLP.

AraBERT: Transformer-based Model for Arabic Language Understanding

Abstract: The Arabic language is a morphologically rich language with relatively few resources and a less explored syntax compared to English. Given these limitations, Arabic Natural Language Processing (NLP) tasks like Sentiment Analysis (SA), Named Entity Recognition (NER), and Question Answering (QA), have proven to be very challenging to tackle. Recently, with the surge of transformers based models, language-specific BERT based models have proven to be very efficient at language understanding, provided they are pre-trained on a very large corpus. Such models were able to set new standards and achieve state-of-the-art results for most NLP tasks. In this paper, we pre-trained BERT specifically for the Arabic language in the pursuit of achieving the same success that BERT did for the English language. The performance of AraBERT is compared to multilingual BERT from Google and other state-of-the-art approaches. The results showed that the newly developed AraBERT achieved state-of-the-art performance on most tested Arabic NLP tasks. The pretrained araBERT models are publicly available on https://github.com/aub-mind/arabert hoping to encourage research and applications for Arabic NLP.

AraGPT2: Pre-Trained Transformer for Arabic Language Generation

Abstract: Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The Mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.

AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding

Abstract: Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.

BERTje: A Dutch BERT Model

Abstract: The transformer-based pre-trained language model BERT has helped to improve state-of-the-art performance on many natural language processing (NLP) tasks. Using the same architecture and parameters, we developed and evaluated a monolingual Dutch BERT model called BERTje. Compared to the multilingual BERT model, which includes Dutch but is only based on Wikipedia text, BERTje is based on a large and diverse dataset of 2.4 billion tokens. BERTje consistently outperforms the equally-sized multilingual BERT model on downstream NLP tasks (part-of-speech tagging, named-entity recognition, semantic role labeling, and sentiment analysis). Our pre-trained Dutch BERT model is made available at https://github.com/wietsedv/bertje.

Multilingual is not enough: BERT for Finnish

Abstract: Deep learning-based language models pretrained on large unannotated text corpora have been demonstrated to allow efficient transfer learning for natural language processing, with recent approaches such as the transformer-based BERT model advancing the state of the art across a variety of tasks. While most work on these models has focused on high-resource languages, in particular English, a number of recent efforts have introduced multilingual models that can be fine-tuned to address tasks in a large number of different languages. However, we still lack a thorough understanding of the capabilities of these models, in particular for lower-resourced languages. In this paper, we focus on Finnish and thoroughly evaluate the multilingual BERT model on a range of tasks, comparing it with a new Finnish BERT model trained from scratch. The new language-specific model is shown to systematically and clearly outperform the multilingual. While the multilingual model largely fails to reach the performance of previously proposed methods, the custom Finnish BERT model establishes new state-of-the-art results on all corpora for all reference tasks: part-of-speech tagging, named entity recognition, and dependency parsing. We release the model and all related resources created for this study with open licenses at this https URL .

Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language

Abstract: The paper introduces methods of adaptation of multilingual masked language models for a specific language. Pre-trained bidirectional language models show state-of-the-art performance on a wide range of tasks including reading comprehension, natural language inference, and sentiment analysis. At the moment there are two alternative approaches to train such models: monolingual and multilingual. While language specific models show superior performance, multilingual models allow to perform a transfer from one language to another and solve tasks for different languages simultaneously. This work shows that transfer learning from a multilingual model to monolingual model results in significant growth of performance on such tasks as reading comprehension, paraphrase detection, and sentiment analysis. Furthermore, multilingual initialization of monolingual model substantially reduces training time. Pre-trained models for the Russian language are open sourced.

Spanish Pre-trained BERT Model and Evaluation Data

Abstract: The Spanish language is one of the top 5 spoken languages in the world. Nevertheless, finding resources to train or evaluate Spanish language models is not an easy task. In this paper we help bridge this gap by presenting a BERT-based language model pre-trained exclusively on Spanish data. As a second contribution, we also compiled several tasks specifically for the Spanish language in a single repository much in the spirit of the GLUE benchmark. By fine-tuning our pre-trained Spanish model, we obtain better results compared to other BERT-based models pre-trained on multilingual corpora for most of the tasks, even achieving a new state-of-the-art on some of them. We have publicly released our model, the pre-training data, and the compilation of the Spanish benchmarks.

CamemBERT: a Tasty French Language Model

Abstract: Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models --in all languages except English-- very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

Abstract: Pre-trained language models (LMs) are currently integral to many natural language processing systems. Although multilingual LMs were also introduced to serve many languages, these have limitations such as being costly at inference time and the size and diversity of non-English data involved in their pre-training. We remedy these issues for a collection of diverse Arabic varieties by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT. To evaluate our models, we also introduce ARLUE, a new benchmark for multi-dialectal Arabic language understanding evaluation. ARLUE is built using 42 datasets targeting six different task clusters, allowing us to offer a series of standardized experiments under rich conditions. When fine-tuned on ARLUE, our models collectively achieve new state-of-the-art results across the majority of tasks (37 out of 48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score (77.40) across all six task clusters, outperforming all other models including XLM-R Large (~ 3.4 x larger size). Our models are publicly available at https://github.com/UBC-NLP/marbert and ARLUE will be released through the same repository.

WangchanBERTa: Pretraining transformer-based Thai Language Models

Abstract: Transformer-based language models, more specifically BERT-based architectures have achieved state-of-the-art performance in many downstream tasks. However, for a relatively low-resource language such as Thai, the choices of models are limited to training a BERT-based model based on a much smaller dataset or finetuning multi-lingual models, both of which yield suboptimal downstream performance. Moreover, large-scale multi-lingual pretraining does not take into account language-specific features for Thai. To overcome these limitations, we pretrain a language model based on RoBERTa-base architecture on a large, deduplicated, cleaned training set (78GB in total size), curated from diverse domains of social media posts, news articles and other publicly available datasets. We apply text processing rules that are specific to Thai most importantly preserving spaces, which are important chunk and sentence boundaries in Thai before subword tokenization. We also experiment with word-level, syllable-level and SentencePiece tokenization with a smaller dataset to explore the effects on tokenization on downstream performance. Our model wangchanberta-base-att-spm-uncased trained on the 78.5GB dataset outperforms strong baselines (NBSVM, CRF and ULMFit) and multi-lingual models (XLMR and mBERT) on both sequence classification and token classification tasks in human-annotated, mono-lingual contexts.

KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding

Abstract: A Lite BERT (ALBERT) has been introduced to scale up deep bidirectional representation learning for natural languages. Due to the lack of pretrained ALBERT models for Korean language, the best available practice is the multilingual model or resorting back to the any other BERT-based model. In this paper, we develop and pretrain KoreALBERT, a monolingual ALBERT model specifically for Korean language understanding. We introduce a new training objective, namely Word Order Prediction (WOP), and use alongside the existing MLM and SOP criteria to the same architecture and model parameters. Despite having significantly fewer model parameters (thus, quicker to train), our pretrained KoreALBERT outperforms its BERT counterpart on 6 different NLU tasks. Consistent with the empirical results in English by Lan et al., KoreALBERT seems to improve downstream task performance involving multi-sentence encoding for Korean language. The pretrained KoreALBERT is publicly available to encourage research and application development for Korean NLP.

Summary of monolingual T-PTLMs.

A summary of the various monolingual model is presented in Tables {{table:ef8bcd93-9167-4fc2-98ae-18a752684b50}}  and {{table:80908966-b37a-4382-b2bb-ad7d717ea6e5}} . Monolingual models are pretrained based on standard model architectures like GPT {{cite:edf566d}} BERT {{cite:b3c7adc}}, {{cite:0a33ffa}}, {{cite:5333341}}, {{cite:268a20b}}, {{cite:27ae3a9}}, {{cite:e38125f}}, {{cite:f67c937}}, {{cite:cead8ff}}, {{cite:6102fa5}}, {{cite:e7bdd06}}, {{cite:7621373}}, {{cite:070d88f}}, {{cite:fb6531b}}, {{cite:cae8df3}}, {{cite:92d7b21}}, {{cite:8fa085d}}, RoBERTa {{cite:cead8ff}}, {{cite:107448c}}, {{cite:b77424f}}, {{cite:ced223a}}, {{cite:0b94237}}, {{cite:b9bfbb6}}, {{cite:a37d01c}}, ALBERT {{cite:5333341}}, {{cite:02e36b4}}, ELECTRA {{cite:2002fd7}}, {{cite:6a22200}}, and T5 {{cite:d239a47}}. As the availability of corpus from one source is limited in the case of many languages, most of these models are pretrained using corpus gathered from multiple sources. For example, IndoBERT {{cite:268a20b}} is pretrained on corpus having text from Wikipedia, News domain, and Internet. Except for models like PTT5 {{cite:d239a47}}, RuBERT {{cite:b3c7adc}} and PortugueseBERT {{cite:0a33ffa}}, all other monolingual models are pretrained from scratch. Models like PTT5, RuBERT, and PortugueseBERT are initialized from existing models and further pretrained with new language-specific vocabulary. In these models, only the transformer encoder layer parameters are copied from existing models while embedding layer parameters are randomly initialized. During CPT, embedding layer parameters are updated along with other layers.


A multi-user MISO system with a dedicated power station

A time division duplex large-scale MIMO system with a co-located energy transmitter and information receiver as well as an information transmitter

Multiple transmitter-receiver pairs with SISO interference channels and a network coordinator

Maximization of the system sum-throughput

Maximization of system energy efficiency

Transmission time constraint and transmit power constraint at the each user and the power station, respectively

Constraints of transmit power, sub-slot duration for RF energy transfer, and information rate

Energy harvesting constraints at receivers

Lagrange multiplier method, an iterative algorithm based on Dinkelbach method {{cite:dd98c32}}

Joint Beamforming Design and Time Allocation for Wireless Powered Communication Networks

Abstract: This paper investigates a multi-input single-output (MISO) wireless powered communication network (WPCN) under the protocol of harvest-then-transmit. The power station (PS) with reliable power supply first replenishes the passive user nodes by radio-frequency wireless power transfer (WPT) in the downlink, then each user node transmits independent information to the sink by a time division multiple access (TDMA) scheme in the uplink. We consider the joint time allocation and beamforming design to maximize the system sum-throughput. With perfect channel station information (CSI) at the PS, a semi-closed form solution is proposed by exploiting the strict concavity of the joint design problem. Then, in the case with Gaussian CSI errors, a robust algorithm is proposed to maximize the system throughput subject to the signal-to-noise ratio (SNR) outage probability constraints. Simulation results demonstrate the efficiency of the proposed fast algorithm and validate the robust algorithm.

Energy-Efficient Optimization for Wireless Information and Power Transfer in Large-Scale MIMO Systems Employing Energy Beamforming

Abstract: In this letter, we consider a large-scale multiple-input multiple-output (MIMO) system where the receiver should harvest energy from the transmitter by wireless power transfer to support its wireless information transmission. The energy beamforming in the large-scale MIMO system is utilized to address the challenging problem of long-distance wireless power transfer. Furthermore, considering the limitation of the power in such a system, this letter focuses on the maximization of the energy efficiency of information transmission (bit per Joule) while satisfying the quality-of-service (QoS) requirement, i.e. delay constraint, by jointly optimizing transfer duration and transmit power. By solving the optimization problem, we derive an energy-efficient resource allocation scheme. Numerical results validate the effectiveness of the proposed scheme.

Collaborative Wireless Energy and Information Transfer in Interference Channel

Abstract: This paper studies the simultaneous wireless information and power transfer (SWIPT) in a multiuser wireless system, in which distributed transmitters send independent messages to their respective receivers, and at the same time cooperatively transmit wireless power to the receivers via energy beamforming. Accordingly, from the wireless information transmission (WIT) perspective, the system of interest can be modeled as the classic interference channel, while it also can be regarded as a distributed multiple-input multiple-output (MIMO) system for collaborative wireless energy transmission (WET). To enable both information decoding (ID) and energy harvesting (EH) in SWIPT, we adopt the low-complexity time switching operation at each receiver to switch between the ID and EH modes over scheduled time. For the hybrid system, we aim to characterize the achievable rate-energy (R-E) trade-offs by various transmitter-side collaboration schemes. Specifically, to facilitate the collaborative energy beamforming, we propose a new signal splitting scheme at the transmitters, where each transmit signal is generally split into an information signal and an energy signal for WIT and WET, respectively. With this new scheme, first, we study the two-user SWIPT system over the fading channel and derive the optimal mode switching rule at the receivers as well as the corresponding transmit signal optimization to achieve various R-E trade-offs. We also compare the R-E performance of our proposed scheme with transmit energy beamforming and signal splitting against two existing schemes with partial or no cooperation of the transmitters. Next, the general case of SWIPT systems with more than two users is studied, for which we propose a practical transmit collaboration scheme by extending the result for the two-user case: we group users into different pairs and apply the cooperation schemes obtained in the two-user case to each paired group. Furthermore, we present a benchmarking scheme based on joint cooperation of all the transmitters inspired by the principle of interference alignment, against which the performance of the proposed scheme is compared.

Summary of Energy Beamforming Designs for Multi-antenna RF-EHNs.

Table {{table:cdb4e211-05e6-42f6-8753-128931bcaf51}}  shows the summary of the existing energy beamforming designs. Compared to SWIPT beamforming, beamforming designs for dedicated energy transfer has been less investigated. Energy beamforming needs to be exploited in more diverse systems, such as heterogeneous networks.


Industrial metal forming anomaly detection

Digital agriculture and smart manufacturing systems anomaly detection

Manufacturing testbeds anomaly detection

Industrial metal (pump) forming anomaly detection

Towards Deep Industrial Transfer Learning for Anomaly Detection on Time Series Data

Abstract: Deep learning promises performant anomaly detection on time-variant datasets, but greatly suffers from low availability of suitable training datasets and frequently changing tasks. Deep transfer learning offers mitigation by letting algorithms built upon previous knowledge from different tasks or locations. In this article, a modular deep learning algorithm for anomaly detection on time series datasets is presented that allows for an easy integration of such transfer learning capabilities. It is thoroughly tested on a dataset from a discrete manufacturing process in order to prove its fundamental adequacy towards deep industrial transfer learning - the transfer of knowledge in industrial applications' special environment.

Time Series Anomaly Detection Using Convolutional Neural Networks and Transfer Learning

Abstract: Time series anomaly detection plays a critical role in automated monitoring systems. Most previous deep learning efforts related to time series anomaly detection were based on recurrent neural networks (RNN). In this paper, we propose a time series segmentation approach based on convolutional neural networks (CNN) for anomaly detection. Moreover, we propose a transfer learning framework that pretrains a model on a large-scale synthetic univariate time series data set and then fine-tunes its weights on small-scale, univariate or multivariate data sets with previously unseen classes of anomalies. For the multivariate case, we introduce a novel network architecture. The approach was tested on multiple synthetic and real data sets successfully.

Robust Detection of Bearing Early Fault Based on Deep Transfer Learning

Abstract: In recent years, machine learning techniques have been proven to be a promising tool for early fault detection of rolling bearings. In many actual applications, however, bearing whole-life data are not easy to be historically accumulated, while insufficient data may result in training a detection model that is not good enough. If utilizing the available data under different working conditions to facilitate model training, the data distribution of different bearings are usually quite different, which does not meet the precondition of i n d e p e n d e n t a n d i d e n t i c a l d i s t r i b u t i o n ( i . i . d ) and tends to cause performance reduction. In addition, disturbed by the unstable noise under complex conditions, most of the current detection methods are inclined to raise false alarms, so that the reliability of detection results needs to be improved. To solve these problems, a robust detection method for bearings early fault is proposed based on deep transfer learning. The method includes offline stage and online stage. In the offline stage, by introducing a deep auto-encoder network with domain adaptation, the distribution inconsistency of normal state data among different bearings can be weakened, then the common feature representation of the normal state is obtained. With the extracted common features, a new state assessment method based on the robust deep auto-encoder network is proposed to evaluate the boundary between normal state and early fault state in the low-rank feature space. By training a support vector machine classifier, the detection model is established. In the online stage, along with the data batch arriving sequentially, the features of target bearing are extracted using the common representation learnt in the offline stage, and online detection is conducted by feeding them into the SVM model. Experimental results on IEEE PHM Challenge 2012 bearing dataset and XJTU-SY dataset show that the proposed approach outperforms several state-of-the-art detection methods in terms of detection accuracy and false alarm rate.

Adaptable and Explainable Predictive Maintenance: Semi-Supervised Deep Learning for Anomaly Detection and Diagnosis in Press Machine Data

Abstract: Predictive maintenance (PdM) has the potential to reduce industrial costs by anticipating failures and extending the work life of components. Nowadays, factories are monitoring their assets and most collected data belong to correct working conditions. Thereby, semi-supervised data-driven models are relevant to enable PdM application by learning from assets’ data. However, their main challenges for application in industry are achieving high accuracy on anomaly detection, diagnosis of novel failures, and adaptability to changing environmental and operational conditions (EOC). This article aims to tackle these challenges, experimenting with algorithms in press machine data of a production line. Initially, state-of-the-art and classic data-driven anomaly detection model performance is compared, including 2D autoencoder, null-space, principal component analysis (PCA), one-class support vector machines (OC-SVM), and extreme learning machine (ELM) algorithms. Then, diagnosis tools are developed supported on autoencoder’s latent space feature vector, including clustering and projection algorithms to cluster data of synthetic failure types semi-supervised. In addition, explainable artificial intelligence techniques have enabled to track the autoencoder’s loss with input data to detect anomalous signals. Finally, transfer learning is applied to adapt autoencoders to changing EOC data of the same process. The data-driven techniques used in this work can be adapted to address other industrial use cases, helping stakeholders gain trust and thus promote the adoption of data-driven PdM systems in smart factories.

Anomaly Detection through Transfer Learning in Agriculture and Manufacturing IoT Systems

Abstract: IoT systems have been facing increasingly sophisticated technical problems due to the growing complexity of these systems and their fast deployment practices. Consequently, IoT managers have to judiciously detect failures (anomalies) in order to reduce their cyber risk and operational cost. While there is a rich literature on anomaly detection in many IoT-based systems, there is no existing work that documents the use of ML models for anomaly detection in digital agriculture and in smart manufacturing systems. These two application domains pose certain salient technical challenges. In agriculture the data is often sparse, due to the vast areas of farms and the requirement to keep the cost of monitoring low. Second, in both domains, there are multiple types of sensors with varying capabilities and costs. The sensor data characteristics change with the operating point of the environment or machines, such as, the RPM of the motor. The inferencing and the anomaly detection processes therefore have to be calibrated for the operating point. In this paper, we analyze data from sensors deployed in an agricultural farm with data from seven different kinds of sensors, and from an advanced manufacturing testbed with vibration sensors. We evaluate the performance of ARIMA and LSTM models for predicting the time series of sensor data. Then, considering the sparse data from one kind of sensor, we perform transfer learning from a high data rate sensor. We then perform anomaly detection using the predicted sensor data. Taken together, we show how in these two application domains, predictive failure classification can be achieved, thus paving the way for predictive maintenance.

Anomaly Detection and Inter-Sensor Transfer Learning on Smart Manufacturing Datasets

Abstract: Smart manufacturing systems are considered the next generation of manufacturing applications. One important goal of the smart manufacturing system is to rapidly detect and anticipate failures to reduce maintenance cost and minimize machine downtime. This often boils down to detecting anomalies within the sensor data acquired from the system which has different characteristics with respect to the operating point of the environment or machines, such as, the RPM of the motor. In this paper, we analyze four datasets from sensors deployed in manufacturing testbeds. We detect the level of defect for each sensor data leveraging deep learning techniques. We also evaluate the performance of several traditional and ML-based forecasting models for predicting the time series of sensor data. We show that careful selection of training data by aggregating multiple predictive RPM values is beneficial. Then, considering the sparse data from one kind of sensor, we perform transfer learning from a high data rate sensor to perform defect type classification. We release our manufacturing database corpus (4 datasets) and codes for anomaly detection and defect type classification for the community to build on it. Taken together, we show that predictive failure classification can be achieved, paving the way for predictive maintenance.

Regularization-based Continual Learning for Anomaly Detection in Discrete Manufacturing

Abstract: The early and robust detection of anomalies occurring in discrete manufacturing processes allows operators to prevent harm, e.g. defects in production machinery or products. While current approaches for data-driven anomaly detection provide good results on the exact processes they were trained on, they often lack the ability to flexibly adapt to changes, e.g. in products. Continual learning promises such flexibility, allowing for an automatic adaption of previously learnt knowledge to new tasks. Therefore, this article discusses different continual learning approaches from the group of regularization strategies, which are implemented, evaluated and compared based on a real industrial metal forming dataset.

Towards Network Traffic Monitoring Using Deep Transfer Learning

Abstract: Network traffic is growing at an outpaced speed globally. The modern network infrastructure makes classic network intrusion detection methods inefficient to classify an inflow of vast network traffic. This paper aims to present a modern approach towards building a network intrusion detection system (NIDS) by using various deep learning methods. To further improve our proposed scheme and make it effective in real-world settings, we use deep transfer learning techniques where we transfer the knowledge learned by our model in a source domain with plentiful computational and data resources to a target domain with sparse availability of both the resources. Our proposed method achieved 98.30% classification accuracy score in the source domain and an improved 98.43% classification accuracy score in the target domain with a boost in the classification speed using UNSW -15 dataset. This study demonstrates that deep transfer learning techniques make it possible to construct large deep learning models to perform network classification, which can be deployed in the real world target domains where they can maintain their classification performance and improve their classification speed despite the limited accessibility of resources.

A compact overview of industrial applications that used deep transfer learning for time series anomaly detection.

Currently, deep transfer learning approaches are popular in the field of computer vision and natural language processing because of the large available datasets.
This popularity is not as pronounced for industrial time series data, likely due to the lack of publicly available data and the domain-specific differences between such data, making the field a less easy target for general gains.
Fortunately, in recent years a growing number of deep transfer learning approaches have been applied in industry to solve anomaly detection tasks, such as fault diagnosis {{cite:18c7cb3}}, quality management {{cite:c410d6b}}, manufacturing process monitoring {{cite:6e706d1}}, network/software security {{cite:e0ba38b}}, and infrastructure monitoring {{cite:81ec5a3}}. These can be mapped onto the core industrial domains of manufacturing process and infrastructure monitoring, predictive maintenance, and energy management. Table {{table:f8a351cf-c2bf-4e03-a8d0-147781729fe1}}  presents a compact comparison of
the related works using deep transfer learning approaches to solve these tasks.


ConceptNet 5.5: An Open Multilingual Graph of General Knowledge

Abstract: Machine learning about language can be improved by supplying it with specific knowledge and sources of external information. We present here a new version of the linked open data resource ConceptNet that is particularly well suited to be used with modern NLP techniques such as word embeddings. ConceptNet is a knowledge graph that connects words and phrases of natural language with labeled edges. Its knowledge is collected from many sources that include expert-created resources, crowd-sourcing, and games with a purpose. It is designed to represent the general knowledge involved in understanding language, improving natural language applications by allowing the application to better understand the meanings behind the words people use. When ConceptNet is combined with word embeddings acquired from distributional semantics (such as word2vec), it provides applications with understanding that they would not acquire from distributional semantics alone, nor from narrower resources such as WordNet or DBPedia. We demonstrate this with state-of-the-art results on intrinsic evaluations of word relatedness that translate into improvements on applications of word vectors, including solving SAT-style analogies.

ERNIE: Enhanced Language Representation with Informative Entities

Abstract: Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The code and datasets will be available in the future.

Open-World Knowledge Graph Completion

Abstract: Knowledge Graphs (KGs) have been applied to many tasks including Web search, link prediction, recommendation, natural language processing, and entity linking. However, most KGs are far from complete and are growing at a rapid pace. To address these problems, Knowledge Graph Completion (KGC) has been proposed to improve KGs by filling in its missing connections. Unlike existing methods which hold a closed-world assumption, i.e., where KGs are fixed and new entities cannot be easily added, in the present work we relax this assumption and propose a new open-world KGC task. As a first attempt to solve this task we introduce an open-world KGC model called ConMask. This model learns embeddings of the entity's name and parts of its text-description to connect unseen entities to the KG. To mitigate the presence of noisy text descriptions, ConMask uses a relationship-dependent content masking to extract relevant snippets and then trains a fully convolutional neural network to fuse the extracted snippets with entities in the KG. Experiments on large data sets, both old and new, show that ConMask performs well in the open-world KGC task and even outperforms existing KGC models on the standard closed-world KGC task.

Adapting Meta Knowledge Graph Information for Multi-Hop Reasoning over Few-Shot Relations

Abstract: Multi-hop knowledge graph (KG) reasoning is an effective and explainable method for predicting the target entity via reasoning paths in query answering (QA) task. Most previous methods assume that every relation in KGs has enough triples for training, regardless of those few-shot relations which cannot provide sufficient triples for training robust reasoning models. In fact, the performance of existing multi-hop reasoning methods drops significantly on few-shot relations. In this paper, we propose a meta-based multi-hop reasoning method (Meta-KGR), which adopts meta-learning to learn effective meta parameters from high-frequency relations that could quickly adapt to few-shot relations. We evaluate Meta-KGR on two public datasets sampled from Freebase and NELL, and the experimental results show that Meta-KGR outperforms state-of-the-art methods in few-shot scenarios. In the future, our codes and datasets will also be available to provide more details.

Overview of most popular tasks in the literature on kg in nlp.

As might be expected, the frequency of occurrence in the literature for the tasks from our taxonomy varies greatly. While Table {{table:3318fb0c-d982-4da0-b5e3-6a1ac169c3e2}}  gives an overview of the most popular tasks, Figure {{figure:2153d1e4-6295-4229-b7ae-d9d3e0dece14}}  compares their popularity over time. Figure {{figure:665f0f05-18ba-4f75-ad47-546506e4f282}}  displays the number of detected domains for the most prominent tasks. It shows that certain tasks are adopted to more domain-specific contexts than others.

Figure {{figure:f4249cdf-bef1-40b2-94a8-b83ba5354afe}}  depicts the different tasks of the analyzed studies and their relative share of contribution types. We can notice that entity extraction and relation extraction, which encompass the most works in line with Table {{table:3318fb0c-d982-4da0-b5e3-6a1ac169c3e2}} , have a very balanced distribution of contribution types. These tasks, which build the foundation for kg construction, have been researched for a long time and the number of studies in these areas is continually increasing, as can be seen in Figure {{figure:2153d1e4-6295-4229-b7ae-d9d3e0dece14}} . Furthermore, a comparison of Figure {{figure:2153d1e4-6295-4229-b7ae-d9d3e0dece14}}  with Figure {{figure:f4249cdf-bef1-40b2-94a8-b83ba5354afe}}  shows that tasks, such as relation extraction or semantic search, which have existed for some time and continue to grow steadily have a rather balanced ratio of contribution types, too. This is an indication that these tasks are already reasonably mature, as some extensive preliminary work is required, for example, to use multiple techniques in a new method.

The frequency of tasks in our survey greatly varies, as reflected in Table {{table:3318fb0c-d982-4da0-b5e3-6a1ac169c3e2}} . Studies concerning kg construction account for the majority of all papers. Applied nlp tasks such as qa and semantic search also have a strong research community. The most emergent topics in recent years have been augmented language models, qa, and kg embedding. Some of the outlined tasks are still confined to the research community, while others have found practical application in many real-life contexts. From Figure {{figure:665f0f05-18ba-4f75-ad47-546506e4f282}}  it is evident that the kg construction tasks and semantic search over kg are the most widely applied ones. Of the nlp tasks, qa and conversational interfaces have been adopted to many real-life domains, usually in the form of digital assistants. Tasks like kg embedding and augmented language models are still only being researched and lack a widespread practical adoption in real-world scenarios. We anticipate that as the research areas of augmented language models and kg embedding mature, more methods and tools will be investigated for these tasks.


Requirements are used to build a virtual environment to test functionalities of a self-driving car

Implementing different algorithms on a dataset to find the effects of sampling rate and house numbers

Capturing unacceptable behaviors and identify the test cases that violate the requirements

Algorithms such as classification and regression performed badly when the sampling rate was low

Data gathered from 3 sensors: CCD camera, lidar, and radar

Data on energy consumption for 3 appliances and 58 houses

Requirements-Driven Test Generation for Autonomous Vehicles With Machine Learning Components

Abstract: Autonomous vehicles are complex systems that are challenging to test and debug. A requirements-driven approach to the development process can decrease the resources required to design and test these systems, while simultaneously increasing the reliability. We present a testing framework that uses signal temporal logic (STL), which is a precise and unambiguous requirements language. Our framework evaluates test cases against the STL formulae and additionally uses the requirements to automatically identify test cases that fail to satisfy the requirements. One of the key features of our tool is the support for machine learning (ML) components in the system design, such as deep neural networks. The framework allows evaluation of the control algorithms, including the ML components, and it also includes models of CCD camera, lidar, and radar sensors, as well as the vehicle environment. We use multiple methods to generate test cases, including covering arrays, which is an efficient method to search discrete variable spaces. The resulting test cases can be used to debug the controller design by identifying controller behaviors that do not satisfy requirements. The test cases can also enhance the testing phase of development by identifying critical corner cases that correspond to the limits of the system's allowed behaviors. We present STL requirements for an autonomous vehicle system, which capture both component-level and system-level behaviors. Additionally, we present three driving scenarios and demonstrate how our requirements-driven testing framework can be used to identify critical system behaviors, which can be used to support the development process.

Data Requirements for Applying Machine Learning to Energy Disaggregation

Abstract: Energy disaggregation, or nonintrusive load monitoring (NILM), is a technology for separating a household’s aggregate electricity consumption information. Although this technology was developed in 1992, its practical usage and mass deployment have been rather limited, possibly because the commonly used datasets are not adequate for NILM research. In this study, we report the findings from a newly collected dataset that contains 10 Hz sampling data for 58 houses. The dataset not only contains the aggregate measurements, but also individual appliance measurements for three types of appliances. By applying three classification algorithms (vanilla DNN (Deep Neural Network), ML (Machine Learning) with feature engineering, and CNN (Convolutional Neural Network) with hyper-parameter tuning) and a recent regression algorithm (Subtask Gated Network) to the new dataset, we show that NILM performance can be significantly limited when the data sampling rate is too low or when the number of distinct houses in the dataset is too small. The well-known NILM datasets that are popular in the research community do not meet these requirements. Our results indicate that higher quality datasets should be used to expedite the progress of NILM research.

Papers with Experiments Research

Eight studies used experiments to evaluate their proposed solutions as shown in Table {{table:cb9cef85-aaec-4481-af00-0b3b0c8d7c77}} . Three studies generated experiments to evaluate a model or method presented {{cite:1b478ce}}, {{cite:23d5c12}}, {{cite:26c07bf}}. Another two studies experimented on finding data requirements. In {{cite:6907c57}} the study focused on finding data requirements for monitoring energy consumption for houses in Japan, and {{cite:98de954}} gathered data to use for training a surgical robot. In the last three experiments,  {{cite:6ec49f9}} evaluates the robustness of requirements for a computer vision system,  {{cite:b000173}} tests requirements for a self-driving car, and {{cite:0a37a52}} experiments on the threshold needed to find the trade-off on the impact of a NFR on the rest.

Survey methods were used as means to gather data for both case studies and experiments. As shown in Tables {{table:4e2758ac-2a50-487d-a9ce-91b7ed0d4b70}}  and {{table:cb9cef85-aaec-4481-af00-0b3b0c8d7c77}} , several studies used surveys to collect data either to build an artefact or to evaluate it. However, we found seven studies that explicitly used surveys as an evaluation technique. These techniques included questionnaires, interviews, workshops, and observations. Table {{table:95b75428-2c85-4ca4-842a-b758d4e47683}}  shows the studies that only performed surveys along with the outcomes. In {{cite:413612e}} the study uses interviews to gather requirements for an orientation and navigation system that is based on computer vision for people who are vision impaired. Lockerbie et al. {{cite:aca031d}} conducts workshops to identify soft goals for an AI application that would provide better quality of life for people with dementia. In {{cite:a8d0482}} the study carried out interviews to identify requirements for a system that flags users with the potential of filing a lawsuit against a given power company. The system also provided information on these customers to the company’s lawyers and approaches such customers to discuss available solutions through a chatbot function. Both studies in  {{cite:89b33a5}}, {{cite:88e4017}} conduct interviews to shed light on the challenges RE faces when building ML systems. And in {{cite:5c343ae}} the authors evaluate a model for a Smart Process Control System through several questionnaires and workshops. Finally, both {{cite:f157413}}, {{cite:59e4404}} use interviews to find requirements for explainable AI.


Autonomous driving in real traffic situations

Autonomous Driving in multiple urban configurations

End to End Learning for Self-Driving Cars

Abstract: We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands. This end-to-end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e.g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS).

ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst

Abstract: Our goal is to train a policy for autonomous driving via imitation learning that is robust enough to drive a real vehicle. We find that standard behavior cloning is insufficient for handling complex driving scenarios, even when we leverage a perception system for preprocessing the input and a controller for executing the output on the car: 30 million examples are still not enough. We propose exposing the learner to synthesized data in the form of perturbations to the expert's driving, which creates interesting situations such as collisions and/or going off the road. Rather than purely imitating all data, we augment the imitation loss with additional losses that penalize undesirable events and encourage progress -- the perturbations then provide an important signal for these losses and lead to robustness of the learned model. We show that the ChauffeurNet model can handle complex situations in simulation, and present ablation experiments that emphasize the importance of each of our proposed changes and show that the model is responding to the appropriate causal factors. Finally, we demonstrate the model driving a car in the real world.

Comparison between model-based and AI-based system architecture for autonomous vehicles.

In the light of the investigated literature, Table {{table:95c3954e-5daf-4039-8362-896ec07c6208}}  illustrates a comparison between performances of the model-based and the AI-based approaches. Further, this table summarized what has been already said in each part through this chapter. Aspects considered in the depicted comparison are the most important requirements in today’s AV in terms of complexity, uncertainty handling or generalization ability. A classification of the AV problem space is proposed in connection with the mentioned requirements.


Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization

Abstract: We introduce “extreme summarization”, a new single-document summarization task which does not favor extractive strategies and calls for an abstractive modeling approach. The idea is to create a short, one-sentence news summary answering the question “What is the article about?”. We collect a real-world, large-scale dataset for this task by harvesting online articles from the British Broadcasting Corporation (BBC). We propose a novel abstractive model which is conditioned on the article’s topics and based entirely on convolutional neural networks. We demonstrate experimentally that this architecture captures long-range dependencies in a document and recognizes pertinent content, outperforming an oracle extractive system and state-of-the-art abstractive approaches when evaluated automatically and by humans.

Teaching Machines to Read and Comprehend

Abstract: Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.

A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents

Abstract: Neural abstractive summarization models have led to promising results in summarizing relatively short documents. We propose the first model for abstractive summarization of single, longer-form documents (e.g., research papers). Our approach consists of a new hierarchical encoder that models the discourse structure of a document, and an attentive discourse-aware decoder to generate the summary. Empirical results on two large-scale datasets of scientific papers show that our model significantly outperforms state-of-the-art models.

WikiHow: A Large Scale Text Summarization Dataset

Abstract: Sequence-to-sequence models have recently gained the state of the art performance in summarization. However, not too many large-scale high-quality datasets are available and almost all the available ones are mainly news articles with specific writing style. Moreover, abstractive human-style systems involving description of the content at a deeper level require data with higher levels of abstraction. In this paper, we present WikiHow, a dataset of more than 230,000 article and summary pairs extracted and constructed from an online knowledge base written by different human authors. The articles span a wide range of topics and therefore represent high diversity styles. We evaluate the performance of the existing methods on WikiHow to present its challenges and set some baselines to further improve it.

SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization

Abstract: This paper introduces the SAMSum Corpus, a new dataset with abstractive dialogue summaries. We investigate the challenges it poses for automated summarization by testing several models and comparing their results with those obtained on a corpus of news articles. We show that model-generated summaries of dialogues achieve higher ROUGE scores than the model-generated summaries of news – in contrast with human evaluators’ judgement. This suggests that a challenging task of abstractive dialogue summarization requires dedicated models and non-standard quality measures. To our knowledge, our study is the first attempt to introduce a high-quality chat-dialogues corpus, manually annotated with abstractive summarizations, which can be used by the research community for further studies.

Semantic Noise Matters for Neural Natural Language Generation

Abstract: Neural natural language generation (NNLG) systems are known for their pathological outputs, i.e. generating text which is unrelated to the input specification. In this paper, we show the impact of semantic noise on state-of-the-art NNLG models which implement different semantic control mechanisms. We find that cleaned data can improve semantic correctness by up to 97%, while maintaining fluency. We also find that the most common error is omitting information, rather than hallucination.

DART: Open-Domain Structured Data Record to Text Generation

Abstract: We present DART, an open domain structured DAta Record to Text generation dataset with over 82k instances (DARTs). Data-to-text annotations can be a costly process, especially when dealing with tables which are the major source of structured data and contain nontrivial structures. To this end, we propose a procedure of extracting semantic triples from tables that encodes their structures by exploiting the semantic dependencies among table headers and the table title. Our dataset construction framework effectively merged heterogeneous sources from open domain semantic parsing and spoken dialogue systems by utilizing techniques including tree ontology annotation, question-answer pair to declarative sentence conversion, and predicate unification, all with minimum post-editing. We present systematic evaluation on DART as well as new state-of-the-art results on WebNLG 2017 to show that DART (1) poses new challenges to existing data-to-text datasets and (2) facilitates out-of-domain generalization. Our data and code can be found at https://github.com/Yale-LILY/dart.

Additional documentation of scientific artifacts used in our paper.

In our experiments, we use several datasets (as detailed above) from public resources . Table {{table:e238fae9-4f3d-495b-a5c9-8ef30413ba2e}}  summarizes the licences. All data are solely used for research purposes.


Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics

Abstract: AI-synthesized face-swapping videos, commonly known as DeepFakes, is an emerging problem threatening the trustworthiness of online information. The need to develop and evaluate DeepFake detection algorithms calls for datasets of DeepFake videos. However, current DeepFake datasets suffer from low visual quality and do not resemble DeepFake videos circulated on the Internet. We present a new large-scale challenging DeepFake video dataset, Celeb-DF, which contains 5,639 high-quality DeepFake videos of celebrities generated using improved synthesis process. We conduct a comprehensive evaluation of DeepFake detection methods and datasets to demonstrate the escalated level of challenges posed by Celeb-DF.

DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection

Abstract: We present our on-going effort of constructing a large- scale benchmark for face forgery detection. The first version of this benchmark, DeeperForensics-1.0, represents the largest face forgery detection dataset by far, with 60, 000 videos constituted by a total of 17.6 million frames, 10 times larger than existing datasets of the same kind. Extensive real-world perturbations are applied to obtain a more challenging benchmark of larger scale and higher diversity. All source videos in DeeperForensics-1.0 are carefully collected, and fake videos are generated by a newly proposed end-to-end face swapping framework. The quality of generated videos outperforms those in existing datasets, validated by user studies. The benchmark features a hidden test set, which contains manipulated videos achieving high deceptive scores in human evaluations. We further contribute a comprehensive study that evaluates five representative detection baselines and make a thorough analysis of different settings.

Summary of the attending datasets in this study. Information includes the publication date, manipulation techniques, number of real and fake videos, and video compression format.

As discussed in section REF  and following the early approach for keyframe extraction {{cite:51cb4d8}}, we first derived all keyframe faces from the videos using FFmpeg {{cite:e97be7a}} and augmentedRandom augmentations include random rotation, transposition, horizontal flipping, vertical flipping, etc. the real keyframe faces for a balanced keyframe dataset. Due to the limited amount, it is insufficient to train the model to converge with keyframes only. Therefore, we further randomly extracted sufficient normal frames from each video. Specifically, to maintain a balanced dataset between the real and fake faces, we extracted three times more normal frame faces from each real video than the fake one. In this study, the proposed model is trained on the FF++ training set and evaluated through the within-dataset test on the FF++ testing set with a mixture of keyframe and normal frame faces. Meanwhile, we collected various popular Deepfake benchmark datasets for extensive experiments on cross-dataset evaluation, namely, Deepfake Detection Challenge (DFDC) {{cite:2810131}}, Celeb-DF {{cite:0298d10}}, and DeeperForensics-1.0 (DF-1.0) {{cite:5d9d5ca}}. A summary of the attending datasets is listed in Table {{table:7ffa7cdd-f343-4189-95d7-27f8e602e58a}} .

Deepfake Detection Challenge (DFDC) {{cite:2810131}} is one of the largest public Deepfake datasets {{cite:f823d8f}}, {{cite:e08a7cb}}, {{cite:5a27a94}} by far with 128,154 videos, which includes synthesized videos using eight different facial manipulation methods as listed in Table {{table:7ffa7cdd-f343-4189-95d7-27f8e602e58a}} . However, its large quantity has caused storage issues in data preprocessing such that the existing approaches have been seldomly attempted on the whole dataset. In this study, we acquired the entire DFDC dataset and randomly shuffled 10% of the whole dataset for cross-dataset evaluation.


Synthetic equity evaluation corpus (EEC)

BERT-based perturbations on natural sentences

On Measuring Gender Bias in Translation of Gender-neutral Pronouns

Abstract: Ethics regarding social bias has recently thrown striking issues in natural language processing. Especially for gender-related topics, the need for a system that reduces the model bias has grown in areas such as image captioning, content recommendation, and automated employment. However, detection and evaluation of gender bias in the machine translation systems are not yet thoroughly investigated, for the task being cross-lingual and challenging to define. In this paper, we propose a scheme for making up a test set that evaluates the gender bias in a machine translation system, with Korean, a language with gender-neutral pronouns. Three word/phrase sets are primarily constructed, each incorporating positive/negative expressions or occupations; all the terms are gender-independent or at least not biased to one side severely. Then, additional sentence lists are constructed concerning formality of the pronouns and politeness of the sentences. With the generated sentence set of size 4,236 in total, we evaluate gender bias in conventional machine translation systems utilizing the proposed measure, which is termed here as translation gender bias index (TGBI). The corpus and the code for evaluation is available on-line.

Automatically Identifying Gender Issues in Machine Translation using Perturbations

Abstract: The successful application of neural methods to machine translation has realized huge quality advances for the community. With these improvements, many have noted outstanding challenges, including the modeling and treatment of gendered language. While previous studies have identified issues using synthetic examples, we develop a novel technique to mine examples from real world data to explore challenges for deployed systems. We use our method to compile an evaluation benchmark spanning examples for four languages from three language families, which we publicly release to facilitate research. The examples in our benchmark expose where model representations are gendered, and the unintended consequences these gendered representations can have in downstream application.

Evaluating Gender Bias in Machine Translation

Abstract: We present the first challenge set and evaluation protocol for the analysis of gender bias in machine translation (MT). Our approach uses two recent coreference resolution datasets composed of English sentences which cast participants into non-stereotypical gender roles (e.g., “The doctor asked the nurse to help her in the operation”). We devise an automatic gender bias evaluation method for eight target languages with grammatical gender, based on morphological analysis (e.g., the use of female inflection for the word “doctor”). Our analyses show that four popular industrial MT systems and two recent state-of-the-art academic MT models are significantly prone to gender-biased translation errors for all tested target languages. Our data and code are publicly available at https://github.com/gabrielStanovsky/mt_gender.

Getting Gender Right in Neural Machine Translation

Abstract: Speakers of different languages must attend to and encode strikingly different aspects of the world in order to use their language correctly (Sapir, 1921; Slobin, 1996). One such difference is related to the way gender is expressed in a language. Saying “I am happy” in English, does not encode any additional knowledge of the speaker that uttered the sentence. However, many other languages do have grammatical gender systems and so such knowledge would be encoded. In order to correctly translate such a sentence into, say, French, the inherent gender information needs to be retained/recovered. The same sentence would become either “Je suis heureux”, for a male speaker or “Je suis heureuse” for a female one. Apart from morphological agreement, demographic factors (gender, age, etc.) also influence our use of language in terms of word choices or syntactic constructions (Tannen, 1991; Pennebaker et al., 2003). We integrate gender information into NMT systems. Our contribution is two-fold: (1) the compilation of large datasets with speaker information for 20 language pairs, and (2) a simple set of experiments that incorporate gender information into NMT for multiple language pairs. Our experiments show that adding a gender feature to an NMT system significantly improves the translation quality for some language pairs.

For eachStudy, the Table shows on which Benchmarkgender bias is assessed,how Gender is intended (here only in binary (b) terms).Finally, we indicate which (R)epresentational – under-representation and stereotyping– or (A)llocational Harm – as reduced quality of service – is addressed in the study.

In the following, we operationalize such categories to map studies on gender bias to their motivations and societal implications (Table {{table:8360b24e-3194-416b-b6c0-885fca2c9d71}}  and {{table:d0296ca2-3eb6-4282-94a3-307cc71ce3d5}} ).

To attest the existence and scale of gender bias across several languages, dedicated benchmarks, evaluations, and experiments have been designed. We first discuss large scale analyses aimed at assessing gender bias in MT, grouped according to two main conceptualizations: i) works focusing on the weight of prejudices and stereotypes in MT ({{formula:f7b1ced4-6960-434f-a8f1-0cfcdb47a364}}REF ); ii) studies assessing whether gender is properly preserved in translation ({{formula:ab53aedf-432b-4433-a905-f38d32a5d84c}}REF ).
In accordance with the human-centered approach embraced in this survey, in Table {{table:8360b24e-3194-416b-b6c0-885fca2c9d71}}  we map each work to the harms (see {{formula:765ba895-48b1-4250-bced-f0ae46a6e5e0}}) ensuing from the biased behaviors they assess. Finally, we review existing benchmarks for comparing MT performance across genders ({{formula:aac5a175-f21a-4b6a-bc92-f0cef7dc2412}}REF ).


Image Labeling on a Network: Using Social-Network Metadata for Image Classification

Abstract: Large-scale image retrieval benchmarks invariably consist of images from the Web. Many of these benchmarks are derived from online photo sharing networks, like Flickr, which in addition to hosting images also provide a highly interactive social community. Such communities generate rich metadata that can naturally be harnessed for image classification and retrieval. Here we study four popular benchmark datasets, extending them with social-network metadata, such as the groups to which each image belongs, the comment thread associated with the image, who uploaded it, their location, and their network of friends. Since these types of data are inherently relational, we propose a model that explicitly accounts for the interdependencies between images sharing common properties. We model the task as a binary labeling problem on a network, and use structured learning techniques to learn model parameters. We find that social-network metadata are useful in a variety of classification tasks, in many cases outperforming methods based on image content.

Open Graph Benchmark: Datasets for Machine Learning on Graphs

Abstract: We present the Open Graph Benchmark (OGB), a diverse set of challenging and realistic benchmark datasets to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale, encompass multiple important graph ML tasks, and cover a diverse range of domains, ranging from social and information networks to biological networks, molecular graphs, source code ASTs, and knowledge graphs. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics. In addition to building the datasets, we also perform extensive benchmark experiments for each dataset. Our experiments suggest that OGB datasets present significant challenges of scalability to large-scale graphs and out-of-distribution generalization under realistic data splits, indicating fruitful opportunities for future research. Finally, OGB provides an automated end-to-end graph ML pipeline that simplifies and standardizes the process of graph data loading, experimental setup, and model evaluation. OGB will be regularly updated and welcomes inputs from the community. OGB datasets as well as data loaders, evaluation scripts, baseline code, and leaderboards are publicly available at https://ogb.stanford.edu .

Inductive Representation Learning on Large Graphs

Abstract: Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches are inherently transductive and do not naturally generalize to unseen nodes. Here we present GraphSAGE, a general, inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data. Instead of training individual embeddings for each node, we learn a function that generates embeddings by sampling and aggregating features from a node's local neighborhood. Our algorithm outperforms strong baselines on three inductive node-classification benchmarks: we classify the category of unseen nodes in evolving information graphs based on citation and Reddit post data, and we show that our algorithm generalizes to completely unseen graphs using a multi-graph dataset of protein-protein interactions.

Statistics of the eight node classification benchmarks.

For a comprehensive comparison, in Section REF  we perform node classification on eight widely-used datasets, covering graphs of various sizes.
The statistics are summarized in Table {{table:a20681c5-16ee-4ccb-bf6f-3af2e364cfed}} .
To evaluate the effectiveness of GraphAKD on graph-level classification, we benchmark GraphAKD against traditional knowledge distillation methods on two molecular property prediction datasets {{cite:b12e2bc}} in Section REF .
For detailed information on the ten datasets, please refer to Appendix .
In a nutshell, all datasets are collected from real-world networks in different domains, including social networks, citation networks, molecular graphs, and trading networks. We conduct experiments under both transductive and inductive settings, involving both textual and visual features.


Market Making via Reinforcement Learning

Abstract: Market making is a fundamental trading problem in which an agent provides liquidity by continually offering to buy and sell a security. The problem is challenging due to inventory risk, the risk of accumulating an unfavourable position and ultimately losing money. In this paper, we develop a high-fidelity simulation of limit order book markets, and use it to design a market making agent using temporal-difference reinforcement learning. We use a linear combination of tile codings as a value function approximator, and design a custom reward function that controls inventory risk. We demonstrate the effectiveness of our approach by showing that our agent outperforms both simple benchmark strategies and a recent online learning approach from the literature.

Robust Market Making via Adversarial Reinforcement Learning

Abstract: We show that adversarial reinforcement learning (ARL) can be used to produce market marking agents that are robust to adversarial and adaptively-chosen market conditions. To apply ARL, we turn the well-studied single-agent model of Avellaneda and Stoikov [2008] into a discrete-time zero-sum game between a market maker and adversary. The adversary acts as a proxy for other market participants that would like to profit at the market maker's expense. We empirically compare two conventional single-agent RL agents with ARL, and show that our ARL approach leads to: 1) the emergence of risk-averse behaviour without constraints or domain-specific penalties; 2) significant improvements in performance across a set of standard metrics, evaluated with or without an adversary in the test environment, and; 3) improved robustness to model uncertainty. We empirically demonstrate that our ARL method consistently converges, and we prove for several special cases that the profiles that we converge to correspond to Nash equilibria in a simplified single-stage game.

Summary of RL for Market Making

Even though market making is a fundamental task in quantitative trading, research on RL-based market making is still at the early stage. Existing few works simply apply different RL methods on their own data. The summary of order execution publications is in Table {{table:a488cc17-efed-4fd9-9119-92399234a144}} . To fully realize the potential of RL for market making, one major obstacle is the lack of high-fidelity micro-level market simulator. At present, there is still no reasonable way to simulate the ubiquitous market impact. This unignorable gap between simulation and real market limits the usage of RL in market making.


A distributed simulation framework for quantum networks and channels

Abstract: We introduce the Simulator for Quantum Networks and Channels ($\texttt{SQUANCH}$), an open-source Python library for creating parallelized simulations of distributed quantum information processing. The framework includes many features of a general-purpose quantum computing simulator, but it is optimized specifically for simulating quantum networks. It includes functionality to allow users to easily design complex multi-party quantum networks, extensible classes for modeling noisy quantum channels, and a multiprocessed NumPy backend for performant simulations. We present an overview of the structure of the library, describing how the various API elements represent the underlying physics and providing simple usage examples for each module. Finally, we present several demonstrations of canonical quantum information protocols implemented using this framework.

SeQUeNCe: A Customizable Discrete-Event Simulator of Quantum Networks

Abstract: Recent advances in quantum information science enabled the development of quantum communication network prototypes and created an opportunity to study full-stack quantum network architectures. This work develops SeQUeNCe, a comprehensive, customizable quantum network simulator. Our simulator consists of five modules: Hardware models, Entanglement Management protocols, Resource Management, Network Management, and Application. This framework is suitable for simulation of quantum network prototypes that capture the breadth of current and future hardware technologies and protocols. We implement a comprehensive suite of network protocols and demonstrate the use of SeQUeNCe by simulating a photonic quantum network with nine routers equipped with quantum memories. The simulation capabilities are illustrated in three use cases. We show the dependence of quantum network throughput on several key hardware parameters and study the impact of classical control message latency. We also investigate quantum memory usage efficiency in routers and demonstrate that redistributing memory according to anticipated load increases network capacity by 69.1% and throughput by 6.8%. We design SeQUeNCe to enable comparisons of alternative quantum network technologies, experiment planning, and validation and to aid with new protocol design. We are releasing SeQUeNCe as an open source tool and aim to generate community interest in extending it.

QuNetSim: A Software Framework for Quantum Networks

Abstract: As quantum internet technologies develop, the need for simulation software and education for quantum internet rises. QuNetSim aims to fill this need. QuNetSim is a Python software framework that can be used to simulate quantum networks up to the network layer. The goal of QuNetSim is to make it easier to investigate and test quantum networking protocols over various quantum network configurations and parameters. The framework incorporates many known quantum network protocols so that users can quickly build simulations and beginners can easily learn to implement their own quantum networking protocols.

NetQASM -- A low-level instruction set architecture for hybrid quantum-classical programs in a quantum internet

Abstract: We introduce NetQASM, a low-level instruction set architecture for quantum internet applications. NetQASM is a universal, platform-independent and extendable instruction set with support for local quantum gates, powerful classical logic and quantum networking operations for remote entanglement generation. Furthermore, NetQASM allows for close integration of classical logic and communication at the application layer with quantum operations at the physical layer. This enables quantum network applications to be programmed in high-level platform-independent software, which is not possible using any other QASM variants. We implement NetQASM in a series of tools to write, parse, encode and run NetQASM code, which are available online. Our tools include a higher-level SDK in Python, which allows an easy way of programming applications for a quantum internet. Our SDK can be used at home by making use of our existing quantum simulators, NetSquid and SimulaQron, and will also provide a public interface to hardware released on a future iteration of Quantum Network Explorer.

Comparison of simulation tools that can be used for designing and evaluating DQC systems.

In Table {{table:23574f37-c94a-4527-8e8a-1bc9fafa40a7}} , we compare some prominent simulation tools that, in our view, can be used for designing and evaluating DQC systems. We propose to classify each tool as belonging to one of three possible classes: i) hardware-oriented (HW), ii) protocol-oriented (PR), and iii) application-oriented (AP). In the remainder of the section, we first present each class with some of the most representative simulation tools. Then, we discuss some open issues.


Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

Remote Sensing Image Scene Classification: Benchmark and State of the Art

Abstract: Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various data sets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning data sets and methods for scene classification is still lacking. In addition, almost all existing data sets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale data set, termed “NWPU-RESISC45,” which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This data set contains 31 500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 1) is large-scale on the scene classes and the total image number; 2) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion; and 3) has high within-class diversity and between-class similarity. The creation of this data set will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed data set, and the results are reported as a useful baseline for future research.

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Abstract: When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover short-comings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Statistics of used datasets.

See Table {{table:ff7e64cb-69ec-4ea9-be1f-8ca9cca1cf54}} . The 1,000 training samples of VTAB-1K are further divided into a training set (800 samples) and a validation set (200 samples) in hyper-parameter tuning. The final results are produced by training on all 1,000 training samples and testing on the test set.