Five NLU tasks (TC, SA, STS, PI, and NLI) with nine datasets and one diagnostic dataset.

Five NLU tasks (QA, WSD, NLI, and Coref) with eight datasets and two diagnostic datasets.

Four NLG tasks (Data2text, TS, TSim, and Dialog) with thirteen data sets.

Eight language generation tasks, including Abstractive TS, Answer-aware Question Generation, Conversational QA, and Personalizing Dialogue.

Seven tweets related tasks, and all are framed as multi-class tweet classification.

Ten programming language understanding and generation tasks with fourteen datasets.

Four NLU tasks (Intent Prediction, Slot-filling, Semantic parsing, and Dialogue state tracking) with seven task-oriented dialogue datasets.

Five tasks (STS, NER, RE, NLI and DC) with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties.

Thirteen biomedical NLP datasets in 6 tasks (NER, PICO, RE, STS, DC, and QA).

Includes tasks like NER, PI, QA, IR, IC, and ToC.

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Abstract: For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems.

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Abstract: In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Abstract: We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for which we are organizing a shared task at our ACL 2021 Workshop and to which we invite the entire NLG community to participate.

GLGE: A New General Language Generation Evaluation Benchmark

Abstract: Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP). These benchmarks mostly focus on a range of Natural Language Understanding (NLU) tasks, without considering the Natural Language Generation (NLG) models. In this paper, we present the General Language Generation Evaluation (GLGE), a new multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. For each task, we continue to design three subtasks in terms of task difficulty (GLGE-Easy, GLGE-Medium, and GLGE-Hard). This introduces 24 subtasks to comprehensively compare model performance. To encourage research on pretraining and transfer learning on NLG models, we make GLGE publicly available and build a leaderboard with strong baselines including MASS, BART, and ProphetNet (The source code and dataset are publicly available at https://github.com/microsoft/glge).

TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification

Abstract: The experimental landscape in natural language processing for social media is too fragmented. Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction. Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domain-specific data. In this paper, we propose a new evaluation framework (TweetEval) consisting of seven heterogeneous Twitter-specific classification tasks. We also provide a strong set of baselines as starting point, and compare different language modeling pre-training strategies. Our initial experiments show the effectiveness of starting off with existing pre-trained generic language models, and continue training them on Twitter corpora.

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Abstract: Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.

DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue

Abstract: A long-standing goal of task-oriented dialogue research is the ability to flexibly adapt dialogue models to new domains. To progress research in this direction, we introduce DialoGLUE (Dialogue Language Understanding Evaluation), a public benchmark consisting of 7 task-oriented dialogue datasets covering 4 distinct natural language understanding tasks, designed to encourage dialogue research in representation-based transfer, domain adaptation, and sample-efficient task learning. We release several strong baseline models, demonstrating performance improvements over a vanilla BERT architecture and state-of-the-art results on 5 out of 7 tasks, by pre-training on a large open-domain dialogue corpus and task-adaptive self-supervised training. Through the DialoGLUE benchmark, the baseline methods, and our evaluation scripts, we hope to facilitate progress towards the goal of developing more general task-oriented dialogue models.

Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

Abstract: Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We make the datasets, pre-trained models, and codes publicly available at https://github.com/ncbi-nlp/BLUE_Benchmark.

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Abstract: Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly-available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition (NER). To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB.

Conceptualized Representation Learning for Chinese Biomedical Text Mining

Abstract: Biomedical text mining is becoming increasingly important as the number of biomedical documents and web data rapidly grows. Recently, word representation models such as BERT has gained popularity among researchers. However, it is difficult to estimate their performance on datasets containing biomedical texts as the word distributions of general and biomedical corpora are quite different. Moreover, the medical domain has long-tail concepts and terminologies that are difficult to be learned via language models. For the Chinese biomedical text, it is more difficult due to its complex structure and the variety of phrase combinations. In this paper, we investigate how the recently introduced pre-trained language model BERT can be adapted for Chinese biomedical corpora and propose a novel conceptualized representation learning approach. We also release a new Chinese Biomedical Language Understanding Evaluation benchmark (\textbf{ChineseBLUE}). We examine the effectiveness of Chinese pre-trained models: BERT, BERT-wwm, RoBERTa, and our approach. Experimental results on the benchmark show that our approach could bring significant gain. We release the pre-trained model on GitHub: this https URL.

Summary of general, social-media and domain-specific Extrinsic Benchmarks. TC - Text Classification, STS- Semantic Text Similarity, TS - Text Summarization, TSim – Text Simplification, ToC – Topic Classification, IC – Intent Classification, IR – Information Retrieval, QA- Question Answering, DC- Document Classification, RE – Relation Extraction, POS - Parts-of-speech tagging, DP - Dependency Parsing, MRC - Machine Reading Comprehension, SA – Sentiment Analysis, PI- Paraphrase Identification, NLI – Natural Language Inference, WPR – Web Page Ranking, QAM – Question Matching, QADSM – Query Ad Matching, MT- Machine Translation, LID – Language Identification, CSR – Common Sense Reasoning, WSD – Word Sense Disambiguation, MCQA – Multiple Choice Question Answering.

Extrinsic evaluation helps to assess the performance of a model in downstream tasks. To get the maximum out of a model, the model should perform well across a wide range of tasks rather than just performing well on one or two tasks. A benchmark provides a standard way of evaluating the model’s generalization ability across tasks. A benchmark usually consists of a set of datasets, a leader board, and a single metric {{cite:d3db81e}}. The datasets are chosen in a way that they are challenging and represent diverse tasks. A leaderboard is an online repository that helps to compare and rank models. For a model to achieve a good score in a benchmark, it should share knowledge i.e., parameters across tasks with one or two layers specific to each task {{cite:d3db81e}}. A benchmark uses a single metric to evaluate the overall performance of the model across tasks. Without a benchmark, it is difficult to evaluate models in a standard way and track the progress in the development of pretrained language models. A summary of various extrinsic bechmarks are presented in Tables {{table:56aeb5d2-5a97-4ab0-be1f-de7f8eca4c29}}  and {{table:4e128f72-9b5b-45c6-8883-c511d8e5fdf8}} .


Eleven tasks in which nine tasks are XNLU (NER, POS, QA, NLI, PI, WPR, QAM, QADSM and TC) and two tasks are XNLG (question and news title generation). This benchmark covers 19 languages.

Five tasks (MT, LID, NER, POS, and SA) with eighteen datasets covering nine different code-switched language pairs.

Eleven datasets covering six tasks (LID, POS, NER, SA, QA and NLI) and two language pairs (English-Hindi and English-Spanish).

Nine tasks spanning forty typologically diverse languages from 12 language families.

Includes ten challenging NLU tasks for 50 languages.

Nine language understanding tasks in Chinese and diagnostic dataset for linguistic analysis.

Twelve tasks clustered into four categories: (a) single-sentence classification, (b) single-sentence sequence tagging, (c) sentence-pair classification, and (d) sentence-pair sequence labelling.

Six commonly used NLG tasks: TS, QA, Chitchat, and three different pairs of machine translation (MT) tasks.

Seven tasks for the Indonesian language spanning Morpho-syntax and Sequence labelling, Semantics, and Discourse with eight datasets.

Six tasks (TC, PI, NLI, WSD, DP, and POS). Three out of six tasks (TC, PI, and NLI) are from cross-lingual datasets.

Six different Arabic language understanding tasks (SA, Social meaning tasks, ToC, Dialect identification, NER, and QA) with 42 datasets.

Seven tasks (NER, Semantic relatedness, QA, TE, SA, and Cyberbully detection) with 9 datasets.

Includes tasks like NLI, PI, NER, POS, DC, DP and QA.

Includes six NLU tasks like TE, PI, SA, MT, MRC, MCQA.

XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation

Abstract: In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in English for natural language understanding tasks only, XGLUE has two main advantages: (1) it provides 11 diversified tasks that cover both natural language understanding and generation scenarios; (2) for each task, it provides labeled data in multiple languages. We extend a recent cross-lingual pre-trained model Unicoder(Huang et al., 2019) to cover both understanding and generation tasks, which is evaluated on XGLUE as a strong baseline. We also evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for comparison.

LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation

Abstract: Recent trends in NLP research have raised an interest in linguistic code-switching (CS); modern approaches have been proposed to solve a wide range of NLP tasks on multiple language pairs. Unfortunately, these proposed methods are hardly generalizable to different code-switched languages. In addition, it is unclear whether a model architecture is applicable for a different task while still being compatible with the code-switching setting. This is mainly because of the lack of a centralized benchmark and the sparse corpora that researchers employ based on their specific needs and interests. To facilitate research in this direction, we propose a centralized benchmark for Linguistic Code-switching Evaluation (LinCE) that combines ten corpora covering four different code-switched language pairs (i.e., Spanish-English, Nepali-English, Hindi-English, and Modern Standard Arabic-Egyptian Arabic) and four tasks (i.e., language identification, named entity recognition, part-of-speech tagging, and sentiment analysis). As part of the benchmark centralization effort, we provide an online platform at ritual.uh.edu/lince, where researchers can submit their results while comparing with others in real-time. In addition, we provide the scores of different popular models, including LSTM, ELMo, and multilingual BERT so that the NLP community can compare against state-of-the-art systems. LinCE is a continuous effort, and we will expand it with more low-resource languages and tasks.

GLUECoS : An Evaluation Benchmark for Code-Switched NLP

Abstract: Code-switching is the use of more than one language in the same conversation or utterance. Recently, multilingual contextual embedding models, trained on multiple monolingual corpora, have shown promising results on cross-lingual and multilingual tasks. We present an evaluation benchmark, GLUECoS, for code-switched languages, that spans several NLP tasks in English-Hindi and English-Spanish. Specifically, our evaluation benchmark includes Language Identification from text, POS tagging, Named Entity Recognition, Sentiment Analysis, Question Answering and a new task for code-switching, Natural Language Inference. We present results on all these tasks using cross-lingual word embedding models and multilingual models. In addition, we fine-tune multilingual models on artificially generated code-switched data. Although multilingual models perform significantly better than cross-lingual models, our results show that in most tasks, across both language pairs, multilingual models fine-tuned on code-switched data perform best, showing that multilingual models can be further optimized for code-switching tasks.

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

Abstract: Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing. To this end, we introduce the Cross-lingual TRansfer Evaluation of Multilingual Encoders XTREME benchmark, a multi-task benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models, particularly on syntactic and sentence retrieval tasks. There is also a wide spread of results across languages. We release the benchmark to encourage research on cross-lingual learning methods that transfer linguistic knowledge across a diverse and representative set of languages and tasks.

XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation

Abstract: Machine learning has brought striking advances in multilingual natural language processing capabilities over the past year. For example, the latest techniques have improved the state-of-the-art performance on the XTREME multilingual benchmark by more than 13 points. While a sizeable gap to human-level performance remains, improvements have been easier to achieve in some tasks than in others. This paper analyzes the current state of cross-lingual transfer learning and summarizes some lessons learned. In order to catalyze meaningful progress, we extend XTREME to XTREME-R, which consists of an improved set of ten natural language understanding tasks, including challenging language-agnostic retrieval tasks, and covers 50 typologically diverse languages. In addition, we provide a massively multilingual diagnostic suite and fine-grained multi-dataset evaluation capabilities through an interactive public leaderboard to gain a better understanding of such models.

RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark

Abstract: In this paper, we introduce an advanced Russian general language understanding evaluation benchmark – Russian SuperGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We also provide baselines, human level evaluation, open-source framework for evaluating models, and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of comparing multilingual models in the translated diagnostic test set and offer the first steps to further expanding or assessing State-of-the-art models independently of language.

CLUE: A Chinese Language Understanding Evaluation Benchmark

Abstract: The advent of natural language understanding (NLU) benchmarks for English, such as GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of tasks. These comprehensive benchmarks have facilitated a broad range of research and applications in natural language processing (NLP). The problem, however, is that most such benchmarks are limited to English, which has made it difficult to replicate many of the successes in English NLU for other languages. To help remedy this issue, we introduce the first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an open-ended, community-driven project that brings together 9 tasks spanning several well-established single-sentence/sentence-pair classification tasks, as well as machine reading comprehension, all on original Chinese text. To establish results on these tasks, we report scores using an exhaustive set of current state-of-the-art pre-trained Chinese models (9 in total). We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on Chinese NLU. Our benchmark is released at https://www.CLUEbenchmarks.com

IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

Abstract: Although Indonesian is known to be the fourth most frequently used language over the internet, the research progress on this language in the natural language processing (NLP) is slow-moving due to a lack of available resources. In response, we introduce the first-ever vast resource for the training, evaluating, and benchmarking on Indonesian natural language understanding (IndoNLU) tasks. IndoNLU includes twelve tasks, ranging from single sentence classification to pair-sentences sequence labeling with different levels of complexity. The datasets for the tasks lie in different domains and styles to ensure task diversity. We also provide a set of Indonesian pre-trained models (IndoBERT) trained from a large and clean Indonesian dataset Indo4B collected from publicly available sources such as social media texts, blogs, news, and websites. We release baseline models for all twelve tasks, as well as the framework for benchmark evaluation, and thus it enables everyone to benchmark their system performances.

IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation

Abstract: Natural language generation (NLG) benchmarks provide an important avenue to measure progress and develop better NLG systems. Unfortunately, the lack of publicly available NLG benchmarks for low-resource languages poses a challenging barrier for building NLG systems that work well for languages with limited amounts of data. Here we introduce IndoNLG, the first benchmark to measure natural language generation (NLG) progress in three low-resource -- yet widely spoken -- languages of Indonesia: Indonesian, Javanese, and Sundanese. Altogether, these languages are spoken by more than 100 million native speakers, and hence constitute an important use case of NLG systems today. Concretely, IndoNLG covers six tasks: summarization, question answering, chit-chat, and three different pairs of machine translation (MT) tasks. We collate a clean pretraining corpus of Indonesian, Sundanese, and Javanese datasets, Indo4B-Plus, which is used to pretrain our models: IndoBART and IndoGPT. We show that IndoBART and IndoGPT achieve competitive performance on all tasks -- despite using only one-fifth the parameters of a larger multilingual model, mBART-LARGE (Liu et al., 2020). This finding emphasizes the importance of pretraining on closely related, local languages to achieve more efficient learning and faster inference for very low-resource languages like Javanese and Sundanese.

IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP

Abstract: Although the Indonesian language is spoken by almost 200 million people and the 10th most spoken language in the world, it is under-represented in NLP research. Previous work on Indonesian has been hampered by a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In this work, we release the IndoLEM dataset comprising seven tasks for the Indonesian language, spanning morpho-syntax, semantics, and discourse. We additionally release IndoBERT, a new pre-trained language model for Indonesian, and evaluate it over IndoLEM, in addition to benchmarking it against existing resources. Our experiments show that IndoBERT achieves state-of-the-art performance over most of the tasks in IndoLEM.

FlauBERT: Unsupervised Language Model Pre-training for French

Abstract: Language models have become a key step to achieve state-of-the art results in many different Natural Language Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their contextualization at the sentence level. This has been widely demonstrated for English using contextualized representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research community for further reproducible experiments in French NLP.

ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

Abstract: Pre-trained language models (LMs) are currently integral to many natural language processing systems. Although multilingual LMs were also introduced to serve many languages, these have limitations such as being costly at inference time and the size and diversity of non-English data involved in their pre-training. We remedy these issues for a collection of diverse Arabic varieties by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT. To evaluate our models, we also introduce ARLUE, a new benchmark for multi-dialectal Arabic language understanding evaluation. ARLUE is built using 42 datasets targeting six different task clusters, allowing us to offer a series of standardized experiments under rich conditions. When fine-tuned on ARLUE, our models collectively achieve new state-of-the-art results across the majority of tasks (37 out of 48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score (77.40) across all six task clusters, outperforming all other models including XLM-R Large (~ 3.4 x larger size). Our models are publicly available at https://github.com/UBC-NLP/marbert and ARLUE will be released through the same repository.

KLEJ: Comprehensive Benchmark for Polish Language Understanding

Abstract: In recent years, a series of Transformer-based models unlocked major improvements in general natural language understanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which allow for a fair comparison of the proposed methods. However, such benchmarks are available only for a handful of languages. To alleviate this issue, we introduce a comprehensive multi-task benchmark for the Polish language understanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing datasets for named entity recognition, question-answering, textual entailment, and others. We also introduce a new sentiment analysis task for the e-commerce domain, named Allegro Reviews (AR). To ensure a common evaluation scheme and promote models that generalize to different NLU tasks, the benchmark includes datasets from varying domains and applications. Additionally, we release HerBERT, a Transformer-based model trained specifically for the Polish language, which has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an extensive evaluation, including several standard baselines and recently proposed, multilingual Transformer-based models.

Spanish Pre-trained BERT Model and Evaluation Data

Abstract: The Spanish language is one of the top 5 spoken languages in the world. Nevertheless, finding resources to train or evaluate Spanish language models is not an easy task. In this paper we help bridge this gap by presenting a BERT-based language model pre-trained exclusively on Spanish data. As a second contribution, we also compiled several tasks specifically for the Spanish language in a single repository much in the spirit of the GLUE benchmark. By fine-tuning our pre-trained Spanish model, we obtain better results compared to other BERT-based models pre-trained on multilingual corpora for most of the tasks, even achieving a new state-of-the-art on some of them. We have publicly released our model, the pre-training data, and the compilation of the Spanish benchmarks.

ParsiNLU: A Suite of Language Understanding Challenges for Persian

Abstract: Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English. This work focuses on Persian language, one of the widely spoken languages in the world, and yet there are few NLU datasets available for this rich language. The availability of high-quality evaluation datasets is a necessity for reliable assessment of the progress on different NLU tasks and domains. We introduce ParsiNLU, the first benchmark in Persian language that includes a range of high-level tasks -- Reading Comprehension, Textual Entailment, etc. These datasets are collected in a multitude of ways, often involving manual annotations by native speakers. This results in over 14.5$k$ new instances across 6 distinct NLU tasks. Besides, we present the first results on state-of-the-art monolingual and multi-lingual pre-trained language-models on this benchmark and compare them with human performance, which provides valuable insights into our ability to tackle natural language understanding challenges in Persian. We hope ParsiNLU fosters further research and advances in Persian language understanding.

Summary of language-based extrinsic benchmarks. TC - Text Classification, STS- Semantic Text Similarity, TS - Text Summarization, TSim – Text Simplification, ToC – Topic Classification, IC – Intent Classification, IR – Information Retrieval, QA- Question Answering, DC- Document Classification, RE – Relation Extraction, POS - Parts-of-speech tagging, DP - Dependency Parsing, MRC - Machine Reading Comprehension, SA – Sentiment Analysis, PI- Paraphrase Identification, NLI – Natural Language Inference, WPR – Web Page Ranking, QAM – Question Matching, QADSM – Query Ad Matching, MT- Machine Translation, LID – Language Identification, CSR – Common Sense Reasoning, WSD – Word Sense Disambiguation, MCQA – Multiple Choice Question Answering.


Joint 2D-3D-Semantic Data for Indoor Scene Understanding

Abstract: We present a dataset of large-scale indoor spaces that provides a variety of mutually registered modalities from 2D, 2.5D and 3D domains, with instance-level semantic and geometric annotations. The dataset covers over 6,000m2 and contains over 70,000 RGB images, along with the corresponding depths, surface normals, semantic annotations, global XYZ images (all in forms of both regular and 360{\deg} equirectangular images) as well as camera information. It also includes registered raw and semantically annotated 3D meshes and point clouds. The dataset enables development of joint and cross-modal learning models and potentially unsupervised approaches utilizing the regularities present in large-scale indoor spaces. The dataset is available here: http://3Dsemantics.stanford.edu/

Semantic Scene Completion from a Single Depth Image

Abstract: This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation. Previous work has considered scene completion and semantic labeling of depth maps separately. However, we observe that these two problems are tightly intertwined. To leverage the coupled nature of these two tasks, we introduce the semantic scene completion network (SSCNet), an end-to-end 3D convolutional network that takes a single depth image as input and simultaneously outputs occupancy and semantic labels for all voxels in the camera view frustum. Our network uses a dilation-based 3D context module to efficiently expand the receptive field and enable 3D context learning. To train our network, we construct SUNCG - a manually created large-scale dataset of synthetic 3D scenes with dense volumetric annotations. Our experiments demonstrate that the joint model outperforms methods addressing each task in isolation and outperforms alternative approaches on the semantic scene completion task.

CHALET: Cornell House Agent Learning Environment

Abstract: We present CHALET, a 3D house simulator with support for navigation and manipulation. CHALET includes 58 rooms and 10 house configuration, and allows to easily create new house and room layouts. CHALET supports a range of common household activities, including moving objects, toggling appliances, and placing objects inside closeable containers. The environment and actions available are designed to create a challenging domain to train and evaluate autonomous agents, including for tasks that combine language, vision, and planning in a dynamic environment.

Matterport3D: Learning from RGB-D Data in Indoor Environments

Abstract: Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.

Gibson Env: Real-World Perception for Embodied Agents

Abstract: Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we are concerned with the problem of developing real-world perception for active agents, propose Gibson Virtual Environment for this purpose, and showcase sample perceptual tasks learned therein. Gibson is based on virtualizing real spaces, rather than using artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism, "Goggles", enabling deploying the trained models in real-world without needing further domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.

The Replica Dataset: A Digital Replica of Indoor Spaces

Abstract: We introduce Replica, a dataset of 18 highly photo-realistic 3D indoor scene reconstructions at room and building scale. Each scene consists of a dense mesh, high-resolution high-dynamic-range (HDR) textures, per-primitive semantic class and instance information, and planar mirror and glass reflectors. The goal of Replica is to enable machine learning (ML) research that relies on visually, geometrically, and semantically realistic generative models of the world - for instance, egocentric computer vision, semantic segmentation in 2D and 3D, geometric inference, and the development of embodied agents (virtual robots) performing navigation, instruction following, and question answering. Due to the high level of realism of the renderings from Replica, there is hope that ML systems trained on Replica may transfer directly to real world image and video data. Together with the data, we are releasing a minimal C++ SDK as a starting point for working with the Replica dataset. In addition, Replica is `Habitat-compatible', i.e. can be natively used with AI Habitat for training and testing embodied agents.

Comparison of existing embodied datasets (*: the datasets render only a room as scene).

An embodied dataset contains 3D assets like textures and meshes for rendering and other configuration data like object location, object category and camera pose for high-level tasks.
A comparison of the proposed datasets is shown in Tab. {{table:545d0f91-b089-4e4b-93ab-f968a7274a5e}} .


Open source equivalent to the WebText corpus used to pretrain GPT-2 model and it is around 32GB.

Collection of hateful comments in English which are posted on Reddit, a popular social media platform.

Consists of de-identified ICU patient records gathered over more than a decade. It is the largest publicly available corpus of clinical records.

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

HateBERT: Retraining BERT for Abusive Language Detection in English

Abstract: We introduce HateBERT, a re-trained BERT model for abusive language detection in English. The model was trained on RAL-E, a large-scale dataset of Reddit comments in English from communities banned for being offensive, abusive, or hateful that we have curated and made available to the public. We present the results of a detailed comparison between a general pre-trained language model and the retrained version on three English datasets for offensive, abusive language and hate speech detection tasks. In all datasets, HateBERT outperforms the corresponding general BERT model. We also discuss a battery of experiments comparing the portability of the fine-tuned models across the datasets, suggesting that portability is affected by compatibility of the annotated phenomena.

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Abstract: Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.

Summary of various pretraining corpora in general, social media and specific domains.

Self-Supervised learning to pretrain T-PTLMs requires large volumes of pretraining data. As shown in Figure, pretraining corpus can be classified into four types (refer Figure {{figure:42747a0c-8b12-4bc3-8813-26ce8b06ddb3}} ). The characteristic of the text differs from one type of corpus to another. For example, in the general domain, the text is less noisy and written formally by professionals. In social media, the text is mostly noisy and written colloquially by the general public. Moreover, many specific domains like Biomedical and Finance contain many domain-specific words which are not used in the general domain. In general, the performance of general domain models in domain-specific tasks is limited {{cite:775cd0a}}. So, we have to choose the pretraining corpus depending on the target domain to achieve good results. BERT model is pretrained using text from Wikipedia and BookCorpus which amounts to 16GB {{cite:2786686}}. Further research works showed that the performance of the model can be increased by using large pretraining datasets {{cite:4a8a477}}, {{cite:1d7dc53}}. This triggered the development of much larger datasets, especially from the common crawl. For example, C4 data contains around 750GB of text data {{cite:f3e91c7}} while CC-100 corpus includes around 2.5TB of text data {{cite:7a4ac22}}. Multilingual T-PTLMs like mBERT {{cite:2786686}}, IndT5 {{cite:f8246e4}}, IndoBART {{cite:d3f39d6}}, and XLM-R {{cite:7a4ac22}} are pretrained using only multilingual datasets. Some of the models like XLM {{cite:39539ef}}, XLM-E {{cite:fa83910}}, infoXLM {{cite:044a7b2}}, and mT6 {{cite:ef49e2b}} are pretrained using both multilingual and parallel datasets. A summary of various pretraining corpora is given in Tables {{table:8e9e64c8-0b1a-4ee0-a0be-5374181e34c9}}  and {{table:baebe5a5-b840-4af7-bf04-3ae4174b5153}} .


Impact of mis primes in probing factual and common-sense knowledge

TREx {{cite:0430bf7}}, GoogleRE, ConceptNet {{cite:ddb3463}} and SQUAD {{cite:1b0e35c}}

TRex {{cite:0430bf7}}, GoogleRE, ConceptNet {{cite:ddb3463}}and SQUAD {{cite:1b0e35c}}

Language Models as Knowledge Bases?

Abstract: Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fill-in-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https://github.com/facebookresearch/LAMA.

Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

Abstract: Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as “Paris is the capital of [MASK]” are used as probes. We translate the established benchmarks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT’s performance as knowledge base language-independent or does it vary from language to language? (iii) A multilingual model is trained on more text, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for better performance? We find that using mBERT as a knowledge base yields varying performance across languages and pooling predictions across languages improves performance. Conversely, mBERT exhibits a language bias; e.g., when queried in Italian, it tends to predict Italy as the country of origin.

Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

Abstract: Building on Petroni et al. 2019, we propose two new probing tasks analyzing factual knowledge stored in Pretrained Language Models (PLMs). (1) Negation. We find that PLMs do not distinguish between negated (‘‘Birds cannot [MASK]”) and non-negated (‘‘Birds can [MASK]”) cloze questions. (2) Mispriming. Inspired by priming methods in human psychology, we add “misprimes” to cloze questions (‘‘Talk? Birds can [MASK]”). We find that PLMs are easily distracted by misprimes. These results suggest that PLMs still have a long way to go to adequately learn human-like factual knowledge.

Does She Wink or Does She Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models

Abstract: Recent progress in pretraining language models on large corpora has resulted in significant performance gains on many NLP tasks. These large models acquire linguistic knowledge during pretraining, which helps to improve performance on downstream tasks via fine-tuning. To assess what kind of knowledge is acquired, language models are commonly probed by querying them with ‘fill in the blank’ style cloze questions. Existing probing datasets mainly focus on knowledge about relations between words and entities. We introduce WDLMPro (Word Definitions Language Model Probing) to evaluate word understanding directly using dictionary definitions of words. In our experiments, three popular pretrained language models struggle to match words and their definitions. This indicates that they understand many words poorly and that our new probing task is a difficult challenge that could help guide research on LMs in the future.

X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models

Abstract: Language models (LMs) have proven surprisingly successful at capturing factual knowledge by completing cloze-style fill-in-the-blank questions such as "Punta Cana is located in _." However, while knowledge is both written and queried in many languages, studies on LMs' factual representation ability have almost invariably been performed on English. To assess factual knowledge retrieval in LMs in different languages, we create a multilingual benchmark of cloze-style probes for 23 typologically diverse languages. To properly handle language variations, we expand probing methods from single- to multi-word entities, and develop several decoding algorithms to generate multi-token predictions. Extensive experimental results provide insights about how well (or poorly) current state-of-the-art LMs perform at this task in languages with more or fewer available resources. We further propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge, and verify its effectiveness on several benchmark languages. Benchmark data and code have been released at https://x-factr.github.io.

Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning

Abstract: Commonsense reasoning research has so far been limited to English. We aim to evaluate and improve popular multilingual language models (ML-LMs) to help advance commonsense reasoning (CSR) beyond English. We collect the Mickey Corpus, consisting of 561k sentences in 11 different languages, which can be used for analyzing and improving ML-LMs. We propose Mickey Probe, a language-agnostic probing task for fairly evaluating the common sense of popular ML-LMs across different languages. In addition, we also create two new datasets, X-CSQA and X-CODAH, by translating their English versions to 15 other languages, so that we can evaluate popular ML-LMs for cross-lingual commonsense reasoning. To improve the performance beyond English, we propose a simple yet effective method -- multilingual contrastive pre-training (MCP). It significantly enhances sentence representations, yielding a large performance gain on both benchmarks.

Summary of various intrinsic benchmarks.

Intrinsic evaluation involves probing the model knowledge using probes like LAMA {{cite:07b5246}}, XLAMA {{cite:4d4972b}}, X-FACTR {{cite:8c23a95}}, MickeyProbe {{cite:25f2ae1}} , Negated LAMA {{cite:07599b0}}, Misprimed LAMA {{cite:07599b0}}, WDLMPro {{cite:6c3cbf6}} or WNLaMPro {{cite:00e4dc1}} (refer Table {{table:b1143522-6db9-4a2a-bcef-67b42d2364b4}} ). LAMA is one of the first probes introduced to evaluate factual and common-sense knowledge in PLTMs under zero-shot settings. LAMA consists of a corpus of facts where the fact can be a relation triplet or a question-answer pair gathered from SQUAD. Here facts are converted to fill-in-the-blank style questions and the model is evaluated based on the prediction of blank tokens i.e., argmax x{{formula:0baafc4b-f915-4a02-ae8a-90e997421f80}}W P(x⁄temp) represents the model vocabulary and {{formula:3c96be93-799e-49f5-be03-80ada71c693f}}  represents the fill-in-the-blank template. LAMA is based on the hypothesis that a model with a good amount of factual knowledge correctly predicts the blank tokens i.e., i.e., the ground truth tokens are predicted with the highest probability compared to other tokens in the model vocabulary. Negated LAMA and Misprimed LAMA probe shows that the language models are not able to consider the negated or misprimed words in the templates. For example, the model predicts the same token whether the templated is negated or not. Poerner et al. (2020) {{cite:50e88fb}} introduced LAMA-UHN which is a collection of triples from the LAMA probing benchmark which are difficult to guess.


image-to-image translation (image artifacts)

pca based outlier removal (image artifacts)

Unsupervised Medical Image Translation Using Cycle-MedGAN

Abstract: Image-to-image translation is a new field in computer vision with multiple potential applications in the medical domain. However, for supervised image translation frameworks, co-registered datasets, paired in a pixel-wise sense, are required. This is often difficult to acquire in realistic medical scenarios. On the other hand, unsupervised translation frameworks often result in blurred translated images with unrealistic details. In this work, we propose a new unsupervised translation framework which is titled Cycle-MedGAN. The proposed framework utilizes new non-adversarial cycle losses which direct the framework to minimize the textural and perceptual discrepancies in the translated images. Qualitative and quantitative comparisons against other unsupervised translation approaches demonstrate the performance of the proposed framework for PET-CT translation and MR motion correction.

PCA leverage: outlier detection for high‐dimensional functional magnetic resonance imaging data

Abstract: &NA; Outlier detection for high‐dimensional (HD) data is a popular topic in modern statistical research. However, one source of HD data that has received relatively little attention is functional magnetic resonance images (fMRI), which consists of hundreds of thousands of measurements sampled at hundreds of time points. At a time when the availability of fMRI data is rapidly growing—primarily through large, publicly available grassroots datasets—automated quality control and outlier detection methods are greatly needed. We propose principal components analysis (PCA) leverage and demonstrate how it can be used to identify outlying time points in an fMRI run. Furthermore, PCA leverage is a measure of the influence of each observation on the estimation of principal components, which are often of interest in fMRI data. We also propose an alternative measure, PCA robust distance, which is less sensitive to outliers and has controllable statistical properties. The proposed methods are validated through simulation studies and are shown to be highly accurate. We also conduct a reliability study using resting‐state fMRI data from the Autism Brain Imaging Data Exchange and find that removal of outliers using the proposed methods results in more reliable estimation of subject‐level resting‐state networks using independent components analysis.

Detecting abnormalities in resting-state dynamics: An unsupervised learning approach

Abstract: Resting-state functional MRI (rs-fMRI) is a rich imaging modality that captures spontaneous brain activity patterns, revealing clues about the connectomic organization of the human brain. While many rs-fMRI studies have focused on static measures of functional connectivity, there has been a recent surge in examining the temporal patterns in these data. In this paper, we explore two strategies for capturing the normal variability in resting-state activity across a healthy population: (a) an autoencoder approach on the rs-fMRI sequence, and (b) a next frame prediction strategy. We show that both approaches can learn useful representations of rs-fMRI data and demonstrate their novel application for abnormality detection in the context of discriminating autism patients from healthy controls.

Table consisting of CT and fMRI image based results of the brain obtained by the literature review

The semi-exhaustive literature review resulted in 45 research items, from which further 6 were removed due to not containing applications in medical imaging (only exemplar stated in abstract) or being non-available. The resulting papers were further clustered into 5 categories (corresponding to Tab. {{table:586d8cd2-d882-4536-a90e-d9e6b792e45b}} -{{table:6b585252-d90d-46db-94b0-1b1db14ac7f6}}  by their imaging methods. Tab. {{table:586d8cd2-d882-4536-a90e-d9e6b792e45b}}  shows papers applying anomaly detection to occular medical images with retinal fundus images and optical coherence tomography (OCT). Tab. {{table:2f8ce732-b2a8-4848-afc9-5dff2b3298fa}}  focuses on papers with applications in the center body region, with chest X-rays and mammography. Tab. {{table:758a0f69-7649-482d-bc91-f5a73ad646b1}}  summarizes application papers, using CT and functional magnetic resonance imaging (fMRI). Tab. {{table:fac491b7-86b6-4824-beb9-d4b963abc220}}  displays papers applying ML to brain Magnetic resonance imaging (MRI) datasets. Tab. {{table:6b585252-d90d-46db-94b0-1b1db14ac7f6}}  shows mixed applications from the domains of breast ultrasound, chest radiographs, histology and fundus images as well as multi-spectral imaging (MSI).


Split learning for health: Distributed deep learning without sharing raw patient data

Abstract: Can health entities collaboratively train deep learning models without sharing sensitive raw data? This paper proposes several configurations of a distributed deep learning method called SplitNN to facilitate such collaborations. SplitNN does not share raw data or model details with collaborating institutions. The proposed configurations of splitNN cater to practical settings of i) entities holding different modalities of patient data, ii) centralized and local health entities collaborating on multiple tasks and iii) learning without sharing labels. We compare performance and resource efficiency trade-offs of splitNN and other distributed deep learning methods like federated learning, large batch synchronous stochastic gradient descent and show highly encouraging results for splitNN.

Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge

Abstract: We envision a mobile edge computing (MEC) framework for machine learning (ML) technologies, which leverages distributed client data and computation resources for training high-performance ML models while preserving client privacy. Toward this future goal, this work aims to extend Federated Learning (FL), a decentralized learning framework that enables privacy-preserving training of models, to work with heterogeneous clients in a practical cellular network. The FL protocol iteratively asks random clients to download a trainable model from a server, update it with own data, and upload the updated model to the server, while asking the server to aggregate multiple client updates to further improve the model. While clients in this protocol are free from disclosing own private data, the overall training process can become inefficient when some clients are with limited computational resources (i.e. requiring longer update time) or under poor wireless channel conditions (longer upload time). Our new FL protocol, which we refer to as FedCS, mitigates this problem and performs FL efficiently while actively managing clients based on their resource conditions. Specifically, FedCS solves a client selection problem with resource constraints, which allows the server to aggregate as many client updates as possible and to accelerate performance improvement in ML models. We conducted an experimental evaluation using publicly-available large-scale image datasets to train deep neural networks on MEC environment simulations. The experimental results show that FedCS is able to complete its training process in a significantly shorter time compared to the original FL protocol.

Detailed comparison of communication efficiency of split learning and federated learning

Abstract: We compare communication efficiencies of two compelling distributed machine learning approaches of split learning and federated learning. We show useful settings under which each method outperforms the other in terms of communication efficiency. We consider various practical scenarios of distributed learning setup and juxtapose the two methods under various real-life scenarios. We consider settings of small and large number of clients as well as small models (1M - 6M parameters), large models (10M - 200M parameters) and very large models (1 Billion-100 Billion parameters). We show that increasing number of clients or increasing model size favors split learning setup over the federated while increasing the number of data samples while keeping the number of clients or model size low makes federated learning more communication efficient.

SplitFed: When Federated Learning Meets Split Learning

Abstract: Federated learning (FL) and split learning (SL) are two popular distributed machine learning approaches. Both follow a model-to-data scenario; clients train and test machine learning models without sharing raw data. SL provides better model privacy than FL due to the machine learning model architecture split between clients and the server. Moreover, the split model makes SL a better option for resource-constrained environments. However, SL performs slower than FL due to the relay-based training across multiple clients. In this regard, this paper presents a novel approach, named splitfed learning (SFL), that amalgamates the two approaches eliminating their inherent drawbacks, along with a refined architectural configuration incorporating differential privacy and PixelDP to enhance data privacy and model robustness. Our analysis and empirical results demonstrate that (pure) SFL provides similar test accuracy and communication efficiency as SL while significantly decreasing its computation time per global epoch than in SL for multiple clients. Furthermore, as in SL, its communication efficiency over FL improves with the number of clients. Besides, the performance of SFL with privacy and robustness measures is further evaluated under extended experimental settings.

Federated Learning for Edge Networks: Resource Optimization and Incentive Mechanism

Abstract: Recent years have witnessed a rapid proliferation of smart Internet of Things (IoT) devices. IoT devices with intelligence require the use of effective machine learning paradigms. Federated learning can be a promising solution for enabling IoT-based smart applications. In this paper, we present the primary design aspects for enabling federated learning at network edge. We model the incentive-based interaction between a global server and participating devices for federated learning via a Stackelberg game to motivate the participation of the devices in the federated learning process. We present several open research challenges with their possible solutions. Finally, we provide an outlook on future research.

Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

Abstract: On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.

Federated Learning with Non-IID Data

Abstract: Federated learning enables resource-constrained edge compute devices, such as mobile phones and IoT devices, to learn a shared model for prediction, while keeping the training data local. This decentralized approach to train models provides privacy, security, regulatory and economic benefits. In this work, we focus on the statistical challenge of federated learning when local data is non-IID. We first show that the accuracy of federated learning reduces significantly, by up to 55% for neural networks trained for highly skewed non-IID data, where each client device trains only on a single class of data. We further show that this accuracy reduction can be explained by the weight divergence, which can be quantified by the earth mover's distance (EMD) between the distribution over classes on each device and the population distribution. As a solution, we propose a strategy to improve training on non-IID data by creating a small subset of data which is globally shared between all the edge devices. Experiments show that accuracy can be increased by 30% for the CIFAR-10 dataset with only 5% globally shared data.

Methodologies for Improving communication efficiency under Edge Heterogeneity

Communication Efficiency Under Edge Heterogeneity

Secondly, for the data distribution, samples held by different clients are typically diverse with different data sizes and distributions, i.e., non-independent and identical distributed (non-IID). For example, in a multi-classification task with {{formula:ee525167-b963-446b-8ba7-93273869f2ba}}  categories. Each Client{{formula:30d24918-c0a6-47d0-b9ab-7ea70f3d5fe2}}  owns a local dataset {{formula:298e1190-1d40-4aff-bf07-b6c5df6726c8}}  that consists of samples with unbalanced labels {{formula:0ba7097d-4155-4a94-b902-ca33b996fe66}} . Then, Client1 has 80% samples from Label1 and Client2 has 80% samples from Label2. Mathematically speaking, suppose that {{formula:d45e86af-7d5f-4af6-b090-11a7d419a06e}}  denotes a supervised neural network classifier with parameters {{formula:c2af8a3d-174c-4a4f-91bf-eddd418abaf8}} , taking an input {{formula:6e4c567d-8fd3-4006-8269-57f646f618d2}}  and outputting a real-valued vector where the {{formula:199f6cc5-ccf8-498c-8fc5-50d7d8237e4e}} th element of the output vector represents the probability that {{formula:72c93d1b-1c72-4fb5-ab74-2c29882240bb}}  is recognized as class {{formula:9a3f0a03-56c0-45bb-905b-87a19d19d1f5}} . Given {{formula:162e240c-7593-4cf1-ae16-ea65c3c86854}} , the prediction is given by {{formula:2164f7ae-2c2b-4ad9-b914-c6de1f763c55}}  where {{formula:ab25752f-2449-4e22-a94f-23a61bb27c90}}  denotes the {{formula:4b4715a4-97ad-4284-8c5f-7a60b2235e39}} th element of {{formula:ea2b746a-aa84-44f1-8633-93bcde9de0da}} . We assume the common data distribution {{formula:cba58c9b-c978-43a5-befa-b6d7867d2041}}  is shared by all clients in FL, and client{{formula:5dd3d638-b31a-4758-8262-32dfcb8e8f89}}  has {{formula:7940d49f-9658-44ac-8cca-f0f2b8461350}} . Then when samples held by clients are skewed with various {{formula:4ce8b994-c4e6-43e0-9ca9-44ae507d19d7}} , {{formula:be2339bf-9af4-41af-b93b-104990444fdb}} , for all {{formula:45ee3ce3-efb1-42bd-b6b8-dcfb394aed3e}} . Client1 follows {{formula:41a05ddc-f568-49fa-9a93-80ca365da2bf}}  and Client2 follows {{formula:d5fdc432-e95b-457f-aa54-538debe4fb72}} , i.e., they are non-IID. Though the random client selection policy in classical FL aims to reduce the time for waiting, the non-IID local data of the selected clients could give rise to a time-consuming convergence or even failing to converge the global model. In this section, we demonstrate the most relevant methodologies used to spread and reduce the amount of data exchanged between the server and clients tackling the edge heterogeneity problem (Table {{table:00794cfb-dba7-4d65-97c1-7c4bb09d2d57}} ).


Towards Automatic Learning of Procedures from Web Instructional Videos

Abstract: The potential for agents, whether embodied or software, to learn by observing other agents performing procedures involving objects and actions is rich. Current research on automatic procedure learning heavily relies on action labels or video subtitles, even during the evaluation phase, which makes them infeasible in real-world scenarios. This leads to our question: can the human-consensus structure of a procedure be learned from a large set of long, unconstrained videos (e.g., instructional videos from YouTube) with only visual evidence? To answer this question, we introduce the problem of procedure segmentation--to segment a video procedure into category-independent procedure segments. Given that no large-scale dataset is available for this problem, we collect a large-scale procedure segmentation dataset with procedure segments temporally localized and described; we use cooking videos and name the dataset YouCook2. We propose a segment-level recurrent network for generating procedure segments by modeling the dependencies across segments. The generated segments can be used as pre-processing for other tasks, such as dense video captioning and event parsing. We show in our experiments that the proposed model outperforms competitive baselines in procedure segmentation.

Dense-Captioning Events in Videos

Abstract: Most natural videos contain numerous events. For example, in a video of a "man playing a piano", the video might also contain "another man dancing" or "a crowd clapping". We introduce the task of dense-captioning events, which involves both detecting and describing events in a video. We propose a new model that is able to identify all events in a single pass of the video while simultaneously describing the detected events with natural language. Our model introduces a variant of an existing proposal module that is designed to capture both short as well as long events that span minutes. To capture the dependencies between the events in a video, our model introduces a new captioning module that uses contextual information from past and future events to jointly describe all events. We also introduce ActivityNet Captions, a large-scale benchmark for dense-captioning events. ActivityNet Captions contains 20k videos amounting to 849 video hours with 100k total descriptions, each with it's unique start and end time. Finally, we report performances of our model for dense-captioning events, video retrieval and localization.

Token statistics for each dataset.

We extract tokens of interest (T.O.I) using the pos-tagger provided by Spacy {{cite:b2359c4}}. In Table {{table:c2a3bf25-0909-46ff-898a-06b23189fc85}} , we show the statistics of tokens for three datasets. For each token that is tagged at VERB or NOUN, we compute the inverse document frequency (idf) by:
{{formula:80479794-73f9-472f-8792-6092137e76ad}}


Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge

Abstract: Gliomas are the most common primary brain malignancies, with different degrees of aggressiveness, variable prognosis and various heterogeneous histologic sub-regions, i.e., peritumoral edematous/invaded tissue, necrotic core, active and non-enhancing core. This intrinsic heterogeneity is also portrayed in their radio-phenotype, as their sub-regions are depicted by varying intensity profiles disseminated across multi-parametric magnetic resonance imaging (mpMRI) scans, reflecting varying biological properties. Their heterogeneous shape, extent, and location are some of the factors that make these tumors difficult to resect, and in some cases inoperable. The amount of resected tumor is a factor also considered in longitudinal scans, when evaluating the apparent tumor for potential diagnosis of progression. Furthermore, there is mounting evidence that accurate segmentation of the various tumor sub-regions can offer the basis for quantitative image analysis towards prediction of patient overall survival. This study assesses the state-of-the-art machine learning (ML) methods used for brain tumor image analysis in mpMRI scans, during the last seven instances of the International Brain Tumor Segmentation (BraTS) challenge, i.e., 2012-2018. Specifically, we focus on i) evaluating segmentations of the various glioma sub-regions in pre-operative mpMRI scans, ii) assessing potential tumor progression by virtue of longitudinal growth of tumor sub-regions, beyond use of the RECIST/RANO criteria, and iii) predicting the overall survival from pre-operative mpMRI scans of patients that underwent gross total resection. Finally, we investigate the challenge of identifying the best ML algorithms for each of these tasks, considering that apart from being diverse on each instance of the challenge, the multi-institutional mpMRI BraTS dataset has also been a continuously evolving/growing dataset.

The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes

Abstract: The morphometry of a kidney tumor revealed by contrast-enhanced Computed Tomography (CT) imaging is an important factor in clinical decision making surrounding the lesion's diagnosis and treatment. Quantitative study of the relationship between kidney tumor morphology and clinical outcomes is difficult due to data scarcity and the laborious nature of manually quantifying imaging predictors. Automatic semantic segmentation of kidneys and kidney tumors is a promising tool towards automatically quantifying a wide array of morphometric features, but no sizeable annotated dataset is currently available to train models for this task. We present the KiTS19 challenge dataset: A collection of multi-phase CT imaging, segmentation masks, and comprehensive clinical outcomes for 300 patients who underwent nephrectomy for kidney tumors at our center between 2010 and 2018. 210 (70%) of these patients were selected at random as the training set for the 2019 MICCAI KiTS Kidney Tumor Segmentation Challenge and have been released publicly. With the presence of clinical context and surgical outcomes, this data can serve not only for benchmarking semantic segmentation models, but also for developing and studying biomarkers which make use of the imaging and semantic segmentation masks.

Dataset summary. C is the number of foreground classes.

We conducted our experiments on Automated Cardiac Diagnosis Challenge 2017 (ACDC17) {{cite:24c0522}}, Multimodal Brain Tumor Segmentation Challenge 2018 (BraTS18) {{cite:ea113b4}} and Kidney Tumor Challenge 2019 (KiTS19) {{cite:80539da}} training sets (see Table {{table:528f343f-fbb1-4814-b350-91f652d87696}}  for a brief overview).
ACDC17 dataset comprised cardiac magnetic resonance images (MRIs) of 100 patients acquired with different equipment.
For each subject, two 3D-image frames were manually segmented into background, right ventricle (RV), myocardium (M) and left ventricle (LV).
BraTS18 dataset included the 210 high-grade glioma patients, and the images were segmented into background, edema (ED), enhancing tumor core (ET) and necrotic and non-enhancing tumor core (NCR_NET).
KiTS19 dataset included 210 patients, and images were segmented into background, kidney and kidney tumor. Additionally, we resized KiTS19 image slices to 256x256 to reduce the computational costs.
The three datasets were divided to training and validation sets, both containing 50% of the subjects, at the subject level so that the training subject set and the validation subject set were mutually exclusive. We standardized all the images to have zero-mean unit-variance.
Note that this split is considerably more challenging than in the segmentation competitions since our training sets are much smaller.


LogoDet-3K: A Large-scale Image Dataset for Logo Detection

Abstract: Logo detection has been gaining considerable attention because of its wide range of applications in the multimedia field, such as copyright infringement detection, brand visibility monitoring, and product brand management on social media. In this article, we introduce LogoDet-3K, the largest logo detection dataset with full annotation, which has 3,000 logo categories, about 200,000 manually annotated logo objects, and 158,652 images. LogoDet-3K creates a more challenging benchmark for logo detection, for its higher comprehensive coverage and wider variety in both logo categories and annotated objects compared with existing datasets. We describe the collection and annotation process of our dataset and analyze its scale and diversity in comparison to other datasets for logo detection. We further propose a strong baseline method Logo-Yolo, which incorporates Focal loss and CIoU loss into the basic YOLOv3 framework for large-scale logo detection. It obtains about 4% improvement on the average performance compared with YOLOv3, and greater improvements compared with reported several deep detection models on LogoDet-3K. We perform extensive evaluation on three other existing datasets to further verify on both logo detection and retrieval tasks, and we demonstrate better generalization ability of LogoDet-3K on logo detection and retrieval tasks. The LogoDet-3K dataset is used to promote large-scale logo-related research. The code and LogoDet-3K can be found at https://github.com/Wangjing1551/LogoDet-3K-Dataset.

Open Logo Detection Challenge

Abstract: Existing logo detection benchmarks consider artificial deployment scenarios by assuming that large training data with fine-grained bounding box annotations for each class are available for model training. Such assumptions are often invalid in realistic logo detection scenarios where new logo classes come progressively and require to be detected with little or none budget for exhaustively labelling fine-grained training data for every new class. Existing benchmarks are thus unable to evaluate the true performance of a logo detection method in realistic and open deployments. In this work, we introduce a more realistic and challenging logo detection setting, called Open Logo Detection. Specifically, this new setting assumes fine-grained labelling only on a small proportion of logo classes whilst the remaining classes have no labelled training data to simulate the open deployment. We further create an open logo detection benchmark, called OpenLogo,to promote the investigation of this new challenge. OpenLogo contains 27,083 images from 352 logo classes, built by aggregating/refining 7 existing datasets and establishing an open logo detection evaluation protocol. To address this challenge, we propose a Context Adversarial Learning (CAL) approach to synthesising training data with coherent logo instance appearance against diverse background context for enabling more effective optimisation of contemporary deep learning detection models. Experiments show the performance advantage of CAL over existing state-of-the-art alternative methods on the more realistic and challenging OpenLogo benchmark.

Statistics of Four Logo Datasets

1) Datasets. We conduct our experiments on four logo datasets with different scales. Most of the experiments are performed on the large-scale LogoDet-3K {{cite:f449e2b}} datasets, which contains 113,710 images for training, 28,432 for validation and 16,510 for testing.
To assess the robustness of the DSFP-GA method, experiments are also performed on the LogoDet-3K-1000 {{cite:f449e2b}} dataset, the middle-scale QMUL-OpenLogo {{cite:9e25e16}} dataset, and the small-scale FlickrLogos-32 {{cite:f1cfdec}} dataset.
The LogoDet-3K-1000 dataset is sampled from the LogoDet-3K dataset, and it consists of 53,049 images for training and 9,559 images for testing.
The QMUL-OpenLogo dataset contains 27,083 images from 352 logo categories by aggregating and refining several existing logo datasets.
The FlickrLogos-32 dataset consists of 2,240 images from 32 logo categories.
The detailed statistics of four datasets are shown in TABLE {{table:e6ecd04f-5d9c-4846-a85a-9a63de666542}} .


Data summary. C = number of foreground classes.

We conducted our experiments on Automated Cardiac Diagnosis Challenge 2017 (ACDC17) {{cite:0b4126b}}, Multimodal Brain Tumor Segmentation Challenge 2018 (BraTS18) {{cite:7e7eaa2}} and Kidney Tumor Challenge 2019 (KiTS19) {{cite:2e84c8a}} training sets (see Table {{table:decc16d8-265c-494a-b9e4-d4f70df01a84}}  for a brief overview).
ACDC17 dataset comprised cardiac magnetic resonance images (MRIs) of 100 patients acquired with different equipment.
For each subject, two 3D-image frames were manually segmented into background, right ventricle (RV), myocardium (M) and left ventricle (LV).
BraTS18 dataset included the 210 high-grade glioma patients, and the images were segmented into background, edema (ED), enhancing tumor core (ET) and necrotic and non-enhancing tumor core (NCR_NET).
KiTS19 dataset included 210 patients, and images were segmented into background, kidney and kidney tumor. Additionally, we resized KiTS19 image slices to 256x256 to reduce the computational costs.
All images were standardized to have zero-mean unit-variance, and we divided each dataset in half: training and validation set.
Note that this split is considerably more challenging than in the segmentation competitions since our training sets are much smaller.


Multitask learning and benchmarking with clinical time series data

Abstract: Health care is one of the most exciting frontiers in data mining and machine learning. Successful adoption of electronic health records (EHRs) created an explosion in digital clinical data available for analysis, but progress in machine learning for healthcare research has been difficult to measure because of the absence of publicly available benchmark data sets. To address this problem, we propose four clinical prediction benchmarks using data derived from the publicly available Medical Information Mart for Intensive Care (MIMIC-III) database. These tasks cover a range of clinical problems including modeling risk of mortality, forecasting length of stay, detecting physiologic decline, and phenotype classification. We propose strong linear and neural baselines for all four tasks and evaluate the effect of deep supervision, multitask training and data-specific architectural modifications on the performance of neural models.

Benchmarking machine learning models on multi-centre eICU critical care dataset

Abstract: Progress of machine learning in critical care has been difficult to track, in part due to absence of public benchmarks. Other fields of research (such as computer vision and natural language processing) have established various competitions and public benchmarks. Recent availability of large clinical datasets has enabled the possibility of establishing public benchmarks. Taking advantage of this opportunity, we propose a public benchmark suite to address four areas of critical care, namely mortality prediction, estimation of length of stay, patient phenotyping and risk of decompensation. We define each task and compare the performance of both clinical models as well as baseline and deep learning models using eICU critical care dataset of around 73,000 patients. This is the first public benchmark on a multi-centre critical care dataset, comparing the performance of clinical gold standard with our predictive model. We also investigate the impact of numerical variables as well as handling of categorical variables on each of the defined tasks. The source code, detailing our methods and experiments is publicly available such that anyone can replicate our results and build upon our work.

Summary of prediction tasks across databases and outcomes

We define five prediction tasks across three electronic health records databases and three outcomes (Table {{table:1b7655a3-c83d-47e6-ae0e-3bd0d5245678}} ), structured in two categories: (1) the prediction of in-hospital mortality, prolonged length of stay, and 30-day readmission upon admission to the hospital and (2) the prediction of in-hospital mortality during the course of a stay in the intensive care unit (ICU).
These tasks are selected for consistency with prior published work {{cite:128a9c3}}, {{cite:37bd7ab}}, {{cite:f4cd404}} and to enable the examination of the generalizability of results across a diverse set of databases containing structured longitudinal electronic health records and temporally-dense intensive care data.


Evaluating Gene Expression in C57BL/6J and DBA/2J Mouse Striatum Using RNA-Seq and Microarrays

Abstract: C57BL/6J (B6) and DBA/2J (D2) are two of the most commonly used inbred mouse strains in neuroscience research. However, the only currently available mouse genome is based entirely on the B6 strain sequence. Subsequently, oligonucleotide microarray probes are based solely on this B6 reference sequence, making their application for gene expression profiling comparisons across mouse strains dubious due to their allelic sequence differences, including single nucleotide polymorphisms (SNPs). The emergence of next-generation sequencing (NGS) and the RNA-Seq application provides a clear alternative to oligonucleotide arrays for detecting differential gene expression without the problems inherent to hybridization-based technologies. Using RNA-Seq, an average of 22 million short sequencing reads were generated per sample for 21 samples (10 B6 and 11 D2), and these reads were aligned to the mouse reference genome, allowing 16,183 Ensembl genes to be queried in striatum for both strains. To determine differential expression, ‘digital mRNA counting’ is applied based on reads that map to exons. The current study compares RNA-Seq (Illumina GA IIx) with two microarray platforms (Illumina MouseRef-8 v2.0 and Affymetrix MOE 430 2.0) to detect differential striatal gene expression between the B6 and D2 inbred mouse strains. We show that by using stringent data processing requirements differential expression as determined by RNA-Seq is concordant with both the Affymetrix and Illumina platforms in more instances than it is concordant with only a single platform, and that instances of discordance with respect to direction of fold change were rare. Finally, we show that additional information is gained from RNA-Seq compared to hybridization-based techniques as RNA-Seq detects more genes than either microarray platform. The majority of genes differentially expressed in RNA-Seq were only detected as present in RNA-Seq, which is important for studies with smaller effect sizes where the sensitivity of hybridization-based techniques could bias interpretation.

RNA-Seq Transcriptome Profiling Identifies CRISPLD2 as a Glucocorticoid Responsive Gene that Modulates Cytokine Function in Airway Smooth Muscle Cells

Abstract: Asthma is a chronic inflammatory respiratory disease that affects over 300 million people worldwide. Glucocorticoids are a mainstay therapy for asthma because they exert anti-inflammatory effects in multiple lung tissues, including the airway smooth muscle (ASM). However, the mechanism by which glucocorticoids suppress inflammation in ASM remains poorly understood. Using RNA-Seq, a high-throughput sequencing method, we characterized transcriptomic changes in four primary human ASM cell lines that were treated with dexamethasone—a potent synthetic glucocorticoid (1 µM for 18 hours). Based on a Benjamini-Hochberg corrected p-value <0.05, we identified 316 differentially expressed genes, including both well known (DUSP1, KLF15, PER1, TSC22D3) and less investigated (C7, CCDC69, CRISPLD2) glucocorticoid-responsive genes. CRISPLD2, which encodes a secreted protein previously implicated in lung development and endotoxin regulation, was found to have SNPs that were moderately associated with inhaled corticosteroid resistance and bronchodilator response among asthma patients in two previously conducted genome-wide association studies. Quantitative RT-PCR and Western blotting showed that dexamethasone treatment significantly increased CRISPLD2 mRNA and protein expression in ASM cells. CRISPLD2 expression was also induced by the inflammatory cytokine IL1β, and small interfering RNA-mediated knockdown of CRISPLD2 further increased IL1β-induced expression of IL6 and IL8. Our findings offer a comprehensive view of the effect of a glucocorticoid on the ASM transcriptome and identify CRISPLD2 as an asthma pharmacogenetics candidate gene that regulates anti-inflammatory effects of glucocorticoids in the ASM.

False Discovery Rate Regression: An Application to Neural Synchrony Detection in Primary Visual Cortex

Abstract: This article introduces false discovery rate regression, a method for incorporating covariate information into large-scale multiple-testing problems. FDR regression estimates a relationship between test-level covariates and the prior probability that a given observation is a signal. It then uses this estimated relationship to inform the outcome of each test in a way that controls the overall false discovery rate at a prespecified level. This poses many subtle issues at the interface between inference and computation, and we investigate several variations of the overall approach. Simulation evidence suggests that: (1) when covariate effects are present, FDR regression improves power for a fixed false-discovery rate; and (2) when covariate effects are absent, the method is robust, in the sense that it does not lead to inflated error rates. We apply the method to neural recordings from primary visual cortex. The goal is to detect pairs of neurons that exhibit fine-time-scale interactions, in the sense that they fire together more often than expected due to chance. Our method detects roughly 50% more synchronous pairs versus a standard FDR-controlling analysis. The companion R package FDRreg implements all methods described in the article. Supplementary materials for this article are available online.

Description of four real datasets. “# tests" shows the number of tests for each dataset after any necessary data pre-processing. DE = Differential Expression.

This section investigates the performance of ZAP using several publicly available real datasets summarized in Table {{table:a792636a-cb6b-4e0e-b242-4c735a25e494}} . Three datasets (bottomly, airway, hippo) are generated by RNA sequencing (RNA-Seq) experiments for detecting differential expressions in transcriptomes, where the primary statistic {{formula:e24a931f-bd85-4fd1-8105-51629b4ed6cd}}  measures the observed difference in the expression level of a gene under two experimental conditions. Meanwhile, an auxiliary covariate, the average normalized read count for each gene, is collected alongside the primary data. The datasets bottomly and airway have been analyzed by the works of {{cite:ef3cd3c}}, {{cite:3317d6d}}, {{cite:b3702bf}} with the methods IHW, AdaPT and CAMT respectively. The more recent data set hippo {{cite:45229cd}} is generated by the cutting-edge single-cell RNA (sc-RNA) sequencing technology to study differential expressions in mouse hippocampus. For all datasets above, we have adopted the standard data pre-processing step, which filters out genes with excessively low read counts across samples before further downstream analyses {{cite:713058e}} such as model fitting and multiple testing. This is a common practice among bioinformaticians for a number of reasons; see Appendix REF  for more discussion. The fourth dataset is based on the experiments in {{cite:a6366dd}} and {{cite:a989872}}, where each {{formula:98a76f39-92a3-4e50-bb43-db1a7680e51a}}  is a normalized test statistic that, for a given pair of neurons in the primary visual cortex, measures how synchronous their spike trains are, and {{cite:2b4cba5}} has applied FDRreg to it for detecting neural interactions. Correspondingly, each such hypothesis has two covariates: the distance and the correlation of the “tuning curves" between the two activated neurons. We have named this dataset scott for short.