Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Describing Textures in the Wild

Abstract: Patterns and textures are defining characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this analytical dimension in image understanding, we address the challenging problem of describing textures with semantic attributes. We identify a rich vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected in the wild.The resulting Describable Textures Dataset (DTD) is the basis to seek for the best texture representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and show that, surprisingly, it outperforms specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that the describable attributes are excellent texture descriptors, transferring between datasets and tasks; in particular, combined with IFV, they significantly outperform the state-of-the-art by more than 8 percent on both FMD and KTHTIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)

Abstract: This work summarizes the results of the largest skin image analysis challenge in the world, hosted by the International Skin Imaging Collaboration (ISIC), a global partnership that has organized the world's largest public repository of dermoscopic images of skin. The challenge was hosted in 2018 at the Medical Image Computing and Computer Assisted Intervention (MICCAI) conference in Granada, Spain. The dataset included over 12,500 images across 3 tasks. 900 users registered for data download, 115 submitted to the lesion segmentation task, 25 submitted to the lesion attribute detection task, and 159 submitted to the disease classification task. Novel evaluation protocols were established, including a new test for segmentation algorithm performance, and a test for algorithm ability to generalize. Results show that top segmentation algorithms still fail on over 10% of images on average, and algorithms with equal performance on test data can have different abilities to generalize. This is an important consideration for agencies regulating the growing set of machine learning tools in the healthcare domain, and sets a new standard for future public challenges in healthcare.

KaoKore: A Pre-modern Japanese Art Facial Expression Dataset

Abstract: From classifying handwritten digits to generating strings of text, the datasets which have received long-time focus from the machine learning community vary greatly in their subject matter. This has motivated a renewed interest in building datasets which are socially and culturally relevant, so that algorithmic research may have a more direct and immediate impact on society. One such area is in history and the humanities, where better and relevant machine learning models can accelerate research across various fields. To this end, newly released benchmarks and models have been proposed for transcribing historical Japanese cursive writing, yet for the field as a whole using machine learning for historical Japanese artworks still remains largely uncharted. To bridge this gap, in this work we propose a new dataset KaoKore which consists of faces extracted from pre-modern Japanese artwork. We demonstrate its value as both a dataset for image classification as well as a creative and artistic dataset, which we explore using generative models. Dataset available at this https URL

Statistics of downstream tranferring datasets, including the train/validation/test split, number of classes and type of visual contents

We present dataset statistics of our downstream transferring task in Table {{table:0e1fe2b9-c1c2-432a-8e6f-93835dcbcf78}} , which contains various types of technical, texture, satellite, natural, medical, illustrative, symbolic and natural visual contents and corresponding categories. For the train/validation/test split of each dataset, we follow the original split for those with official split file, i.e. Aircraft, DTD (the first official split), Flowers and Kaokore. For the remaining datasets with only train/test split, we preserve the test set, and randomly split the training set into training and validation sets with the proportion of {{formula:c5722703-2628-4037-b008-4811aa2f3a37}}  inside each category.


Tasks: DetectionFree and open source detector based on deep learning hosted by Microsoft AI4Earth. The current model is
trained with the TensorFlow Object Detection API using several hundred thousand camera trap images labeled with bounding boxes from a variety of ecosystems. The model identifies animals (not species-specific), humans, and vehicles, and is robust to novel sensor deployment locations and taxa not seen during training. Updates of the model, trained with additional data, are periodically released. Microsoft AI4Earth provides support to assist ecologists in using the model, including a public API for batch inference, and integration with commonly-used camera trap data management platforms such as TimeLapse and Camelot.

Tasks: Pose estimation and behavioral analysis.Free and open source pose estimation toolbox based on deep learning. Pre-trained models (for instance for primate faces and bodies, as well as quadruped) as well as a light-weight, real-time version are available.

Efficient Pipeline for Camera Trap Image Review

Abstract: Biologists all over the world use camera traps to monitor biodiversity and wildlife population density. The computer vision community has been making strides towards automating the species classification challenge in camera traps, but it has proven difficult to to apply models trained in one region to images collected in different geographic areas. In some cases, accuracy falls off catastrophically in new region, due to both changes in background and the presence of previously-unseen species. We propose a pipeline that takes advantage of a pre-trained general animal detector and a smaller set of labeled images to train a classification model that can efficiently achieve accurate results in a new region.

Markerless tracking of user-defined features with deep learning

Abstract: Quantifying behavior is crucial for many applications in neuroscience. Videography provides easy methods for the observation and recording of animal behavior in diverse settings, yet extracting particular aspects of a behavior for further analysis can be highly time consuming. In motor control studies, humans or other animals are often marked with reflective markers to assist with computer-based tracking, yet markers are intrusive (especially for smaller animals), and the number and location of the markers must be determined a priori. Here, we present a highly efficient method for markerless tracking based on transfer learning with deep neural networks that achieves excellent results with minimal training data. We demonstrate the versatility of this framework by tracking various body parts in a broad collection of experimental settings: mice odor trail-tracking, egg-laying behavior in drosophila, and mouse hand articulation in a skilled forelimb task. For example, during the skilled reaching behavior, individual joints can be automatically tracked (and a confidence score is reported). Remarkably, even when a small number of frames are labeled ($\approx 200$), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy.

Resources for machine and deep learning-based wildlife conservation

Technology to accelerate ecology and conservation research

Machine learning (ML, see glossary in Supplement) deals with learning patterns from data {{cite:1bc2165}}. Presented with large quantities of inputs (e.g., images) and corresponding expected outcomes, or labels (e.g., the species depicted in each image), a supervised ML algorithm learns a mathematical function leading to the correct outcome prediction when confronted with new, unseen inputs. When the expected outcomes are absent, the (this time unsupervised) ML algorithm will use solely the inputs to extract groups of data points corresponding to typical patterns in the data. ML has emerged as a promising means of connecting the dots between big data and actionable ecological insights {{cite:ca38121}} and is an increasingly popular approach in ecology {{cite:d9ddcbd}}, {{cite:6fb1ab5}}, especially due to increased computational power and rapid software advances of the last ten years. A significant share of this success can be attributed to deep learning (DL {{cite:946b43a}}), a family of highly versatile ML models based on artificial neural networks that have shown superior performance across the majority of ML use cases (see Table {{table:08f59ec6-3d56-4c45-a402-d69071b6da9a}}  and Supplement). Significant error reduction of ML and DL with respect to traditional generalized regression models has been reported routinely for species richness and diversity estimation {{cite:1adf342}}, {{cite:c0c77b6}}. Likewise, detection and counting pipelines moved from rough rule of thumb extrapolations from visual counts in national parks to ML-based methods with high detection rates. Initially, these methods proposed many false positives which required further human review {{cite:207e672}}, but recent methods have been shown to maintain high detection rates with significantly fewer false positives {{cite:2b48cc2}}. As an example, large mammal detection in the Kuzikus reserve in 2014 was improved significantly by improving the detection methodologies, from a recall rate of 20% {{cite:207e672}}, to 80% {{cite:5c42f57}} (for a common 75% precision rate). Finally, studies involving human operators demonstrated that ML enabled massive speedups in complex tasks such as individual and species recognition {{cite:fdb4b51}}, {{cite:d366263}} and large-scale tasks such as animal detection in drone surveys {{cite:510d79d}}.
Recent advances in ML methodology could accelerate and enhance various stages of the traditional ecological research pipeline (see Figure {{figure:06e5a373-436c-4362-9a5c-bd3e2be336fe}} ), from targeted data acquisition to image retrieval and semi-automated population surveys. As an example, the initiative Wildlife Insights {{cite:8c84737}} is now processing millions of camera trap images automatically (17 million in August 2021), providing wildlife conservation scientists and practitioners with the data necessary to study animal abundances, diversity and behavior. Besides pure acceleration, use of ML also massively reduces analysis costs, with reduction factors estimated between 2 and 10 {{cite:f60b064}}.

Community science for crowd-sourcing data

A particularly interesting case is Wildbook (see the text box on Page {{figure:bc4d2c0c-d2f1-43e8-ac62-f6b98ce03e02}}  and Table {{table:08f59ec6-3d56-4c45-a402-d69071b6da9a}} ), which routinely screens videos from social media platforms with computer vision models to identify individuals; community members (in this case video posters) are then queried in case of missing or uncertain information. Recent research shows that ML models trained on community data can perform as well as annotators {{cite:da214e4}}. However, it is prudent to note that the viability of community science services may be limited depending on the task and that oftentimes substantial efforts are required to verify volunteer-annotated data. This is due to annotator errors, including misdetected or mislabeled animals due to annotator fatigue or insufficient knowledge about the annotation task, as well as systematic errors from adversarial annotators.

Wildlife detection and species-level classification

Conservation efforts of endangered species require knowledge on how many individuals of the species in question are present in a study area. Such estimations are conventionally realized with statistical occurrence models that are informed by sample-based species observations. It is these observations where imaging sensors (camera traps, UAVs, etc.), paired with ML models that detect and count individuals in the imagery, can provide the most significant input. Early works used classical supervised ML algorithms (algorithms needing a set of human-labeled annotations to learn, see Supplementary Table 2): these algorithms were used to make the connection between a set of characteristics of interest extracted from the image (visual descriptors, e.g., color histograms, spectral indices, etc., also see the glossary) and the annotation itself (presence of an animal, species, etc.) {{cite:fe4c8f1}}, {{cite:207e672}}. Particularly in camera trap imagery, foreground (animal) segmentation is occasionally performed as a pre-processing step to discard image parts that are potentially confusing for a classifier. These approaches, albeit good in performance, suffer from two limitations: first, the visual descriptors need to be specifically tailored to the problem and dataset at hand. This not only requires significant engineering efforts, but also bears the risk of the model becoming too specific to the particular dataset and external conditions (e.g., camera type, background foliage amount and movement type) at hand. Secondly, computational restrictions in these models limit the number of training examples, which is likely to have detrimental effects on variations in data (temporal, seasonal, etc.), thus reducing the generalization capabilities to new sensor deployments or regions. For these reasons, detecting and classifying animal species with DL for the purpose of population estimates is becoming increasingly popular for images {{cite:cfb5dc7}}, {{cite:7b1ffd6}}, acoustic spectrograms {{cite:cf8e74c}}, and videos {{cite:d33069f}}. Models performing accurately and robustly on specific classes (e.g. the MegaDetector or AIDE; see Table {{table:08f59ec6-3d56-4c45-a402-d69071b6da9a}} ) are now being used routinely and integrated within open systems supporting ecologists performing both labeling and detection, respectively counting of their image databases.
Issues related to dependence of the models performance to specific training locations are still at the core of current developments {{cite:7b1ffd6}}, an issue known in ML as “domain adaptation” or “generalization”.


Pilot Assignment in Cell-Free Massive MIMO based on the Hungarian Algorithm

Abstract: This letter focuses on the problem of pilot assignment in cell-free massive MIMO systems. Exploiting the well-known Hungarian algorithms, two procedures are proposed, one maximizing the system throughput, and the other one maximizing the fairness across users. The algorithms operate based on the knowledge of large-scale fading coefficients as a proxy for the distances between users in the system and take into account both the uplink and downlink performance. Numerical results show that the proposed pilot assignment algorithms are effective and outperform the many competing alternatives available in the literature.

Making Cell-Free Massive MIMO Competitive With MMSE Processing and Centralized Implementation

Abstract: Cell-free Massive MIMO is considered as a promising technology for satisfying the increasing number of users and high rate expectations in beyond-5G networks. The key idea is to let many distributed access points (APs) communicate with all users in the network, possibly by using joint coherent signal processing. The aim of this paper is to provide the first comprehensive analysis of this technology under different degrees of cooperation among the APs. Particularly, the uplink spectral efficiencies of four different cell-free implementations are analyzed, with spatially correlated fading and arbitrary linear processing. It turns out that it is possible to outperform conventional Cellular Massive MIMO and small cell networks by a wide margin, but only using global or local minimum mean-square error (MMSE) combining. This is in sharp contrast to the existing literature, which advocates for maximum-ratio combining. Also, we show that a centralized implementation with optimal MMSE processing not only maximizes the SE but largely reduces the fronthaul signaling compared to the standard distributed approach. This makes it the preferred way to operate Cell-free Massive MIMO networks. Non-linear decoding is also investigated and shown to bring negligible improvements.

Cell-Free Massive MIMO versus Small Cells

Abstract: A Cell-Free Massive MIMO (multiple-input multiple-output) system comprises a very large number of distributed access points (APs)which simultaneously serve a much smaller number of users over the same time/frequency resources based on directly measured channel characteristics. The APs and users have only one antenna each. The APs acquire channel state information through time-division duplex operation and the reception of uplink pilot signals transmitted by the users. The APs perform multiplexing/de-multiplexing through conjugate beamforming on the downlink and matched filtering on the uplink. Closed-form expressions for individual user uplink and downlink throughputs lead to max-min power control algorithms. Max-min power control ensures uniformly good service throughout the area of coverage. A pilot assignment algorithm helps to mitigate the effects of pilot contamination, but power control is far more important in that regard. Cell-Free Massive MIMO has considerably improved performance with respect to a conventional small-cell scheme, whereby each user is served by a dedicated AP, in terms of both 95%-likely per-user throughput and immunity to shadow fading spatial correlation. Under uncorrelated shadow fading conditions, the cell-free scheme provides nearly 5-fold improvement in 95%-likely per-user throughput over the small-cell scheme, and 10-fold improvement when shadow fading is correlated.

MMSE-Optimal Sequential Processing for Cell-Free Massive MIMO With Radio Stripes

Abstract: Cell-free massive multiple-input-multiple-output (mMIMO) is an emerging technology for beyond 5G with its promising features such as higher spectral efficiency and superior spatial diversity as compared to conventional multiple-input-multiple-output (MIMO) technology. The main working principle of cell-free mMIMO is that many distributed access points (APs) cooperate simultaneously to serve all the users within the network without creating cell boundaries. This paper considers the uplink of a cell-free mMIMO system utilizing the radio stripe network architecture with a sequential fronthaul between the APs. A novel uplink sequential processing algorithm is developed which is proved to be optimal in both the maximum spectral efficiency (SE) and the minimum mean square error (MSE) sense. A detailed quantitative analysis of the fronthaul requirement or signaling of the proposed algorithm and its comparison with competing sub-optimal algorithms is provided. Key conclusions and implications are summarized in the form of corollaries. Based on the analytical and numerical simulation results, we conclude that the proposed scheme can greatly reduce the fronthaul signaling, without compromising the communication performance.

Distributed Expectation Propagation Detection for Cell-Free Massive MIMO

Abstract: In cell-free massive MIMO networks, an efficient distributed detection algorithm is of significant importance. In this paper, we propose a distributed expectation propagation (EP) detector for cell-free massive MIMO. The detector is composed of two modules, a nonlinear module at the central processing unit (CPU) and a linear module at the access point (AP). The turbo principle in iterative decoding is utilized to compute and pass the extrinsic information between modules. An analytical framework is then provided to characterize the asymptotic performance of the proposed EP detector with a large number of antennas. Simulation results will show that the proposed method outperforms the distributed detectors in terms of bit-error-rate.

Enhanced Normalized Conjugate Beamforming for Cell-Free Massive MIMO

Abstract: In cell-free massive multiple-input multiple-output (MIMO) the fluctuations of the channel gain from the access points to a user are large due to the distributed topology of the system. Because of these fluctuations, data decoding schemes that treat the channel as deterministic perform inefficiently. A way to reduce the channel fluctuations is to design a precoding scheme that equalizes the effective channel gain seen by the users. Conjugate beamforming (CB) poorly contributes to harden the effective channel at the users. In this work, we propose a variant of CB dubbed enhanced normalized CB (ECB), in that the precoding vector consists of the conjugate of the channel estimate normalized by its squared norm. For this scheme, we derive an exact closed-form expression for an achievable downlink spectral efficiency (SE), accounting for channel estimation errors, pilot reuse and user's lack of channel state information (CSI), assuming independent Rayleigh fading channels. We also devise an optimal max-min fairness power allocation based only on large-scale fading quantities. ECB greatly boosts the channel hardening enabling the users to reliably decode data relying only on statistical CSI. As the provided effective channel is nearly deterministic, acquiring CSI at the users does not yield a significant gain.

Team MMSE Precoding with Applications to Cell-free Massive MIMO

Abstract: This article studies a novel distributed precoding design, coined team minimum mean-square error (TMMSE) precoding, which rigorously generalizes classical centralized MMSE precoding to distributed operations based on transmitter-specific channel state information (CSIT). Building on the so-called theory of teams, we derive a set of necessary and sufficient conditions for optimal TMMSE precoding, in the form of an infinite dimensional linear system of equations. These optimality conditions are further specialized to cell-free massive MIMO networks, and explicitly solved for two important examples, i.e., the classical case of local CSIT and the case of unidirectional CSIT sharing along a serial fronthaul. The latter case is relevant, e.g., for the recently proposed radio stripe concept and the related advances on sequential processing exploiting serial connections. In both cases, our optimal design outperforms the heuristic methods that are known from the previous literature. Duality arguments and numerical simulations validate the effectiveness of the proposed team theoretical approach in terms of ergodic achievable rates under a sum-power constraint.

Distributed Precoding Design via Over-the-Air Signaling for Cell-Free Massive MIMO

Abstract: Most works on cell-free massive multiple-input multiple-output (MIMO) consider non-cooperative precoding strategies at the base stations (BSs) to avoid extensive channel state information (CSI) exchange via backhaul signaling. However, considerable performance gains can be accomplished by allowing coordination among the BSs. This paper proposes the first distributed framework for cooperative precoding design in cell-free massive MIMO (and, more generally, in joint transmission coordinated multi-point) systems that entirely eliminates the need for backhaul signaling for CSI exchange. A novel over-the-air (OTA) signaling mechanism is introduced such that each BS can obtain the same cross-term information that is traditionally exchanged among the BSs via backhaul signaling. The proposed distributed precoding design enjoys desirable flexibility and scalability properties, as the amount of OTA signaling does not scale with the number of BSs or user equipments. Numerical results show fast convergence and remarkable performance gains as compared with non-cooperative precoding design. The proposed scheme can also outperform the centralized precoding design under realistic CSI acquisition.

Cell-Free Massive MIMO for Wireless Federated Learning

Abstract: This paper proposes a novel scheme for cell-free massive multiple-input multiple-output (CFmMIMO) networks to support any federated learning (FL) framework. This scheme allows each instead of all the iterations of the FL framework to happen in a large-scale coherence time to guarantee a stable operation of an FL process. To show how to optimize the FL performance using this proposed scheme, we consider an existing FL framework as an example and target FL training time minimization for this framework. An optimization problem is then formulated to jointly optimize the local accuracy, transmit power, data rate, and users' processing frequency. This mixed-timescale stochastic nonconvex problem captures the complex interactions among the training time, and transmission and computation of training updates of one FL process. By employing the online successive convex approximation approach, we develop a new algorithm to solve the formulated problem with proven convergence to the neighbourhood of its stationary points. Our numerical results confirm that the presented joint design reduces the training time by up to $55\%$ over baseline approaches. They also show that CFmMIMO here requires the lowest training time for FL processes compared with cell-free time-division multiple access massive MIMO and collocated massive MIMO.

Max-Min Rate of Cell-Free Massive MIMO Uplink with Optimal Uniform Quantization

Abstract: Cell-free Massive multiple-input multiple-output (MIMO) is considered, where distributed access points (APs) multiply the received signal by the conjugate of the estimated channel, and send back a quantized version of this weighted signal to a central processing unit (CPU). For the first time, we present a performance comparison between the case of perfect fronthaul links, the case when the quantized version of the estimated channel and the quantized signal are available at the CPU, and the case when only the quantized weighted signal is available at the CPU. The Bussgang decomposition is used to model the effect of quantization. The max-min problem is studied, where the minimum rate is maximized with the power and fronthaul capacity constraints. To deal with the non-convex problem, the original problem is decomposed into two sub-problems (referred to as receiver filter design and power allocation). Geometric programming (GP) is exploited to solve the power allocation problem whereas a generalized eigenvalue problem is solved to design the receiver filter. An iterative scheme is developed and the optimality of the proposed algorithm is proved through uplink-downlink duality. A user assignment algorithm is proposed which significantly improves the performance. Numerical results demonstrate the superiority of the proposed schemes.

.  Overview of different signal processing algorithms for cell-free massive MIMO systems

The remaining part of this paper is organized as follows. Section  first introduces the system model of cell-free massive MIMO. Section  reviews existing signal processing algorithms in cell-free massive MIMO physical layer. Section  discusses some open directions in this area and our conclusions are given in Section . The key topics and associated references are summarized in Table {{table:de07c2aa-79ba-4b83-bdb3-72b539a35e37}}  and the definition of the main mathematical symbols is presented in Table {{table:4b45b125-1134-44cd-9d48-ffd547dc1b94}} .


Deep Learning based Vulnerability Detection: Are We There Yet?

Abstract: Automated detection of software vulnerabilities is a fundamental problem in software security. Existing program analysis techniques either suffer from high false positives or false negatives. Recent progress in Deep Learning (DL) has resulted in a surge of interest in applying DL for automated vulnerability detection. Several recent studies have demonstrated promising results achieving an accuracy of up to 95% at detecting vulnerabilities. In this paper, we ask, "how well do the state-of-the-art DL-based techniques perform in a real-world vulnerability prediction scenario?". To our surprise, we find that their performance drops by more than 50%. A systematic investigation of what causes such precipitous performance drop reveals that existing DL-based vulnerability prediction approaches suffer from challenges with the training data (e.g., data duplication, unrealistic distribution of vulnerable classes, etc.) and with the model choices (e.g., simple token-based models). As a result, these approaches often do not learn features related to the actual cause of the vulnerabilities. Instead, they learn unrelated artifacts from the dataset (e.g., specific variable/function names, etc.). Leveraging these empirical findings, we demonstrate how a more principled approach to data collection and model design, based on realistic settings of vulnerability prediction, can lead to better solutions. The resulting tools perform significantly better than the studied baseline: up to 33.57% boost in precision and 128.38% boost in recall compared to the best performing model in the literature. Overall, this paper elucidates existing DL-based vulnerability prediction systems' potential issues and draws a roadmap for future DL-based vulnerability prediction research. In that spirit, we make available all the artifacts supporting our results: https://git.io/Jf6IA.

Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks

Abstract: Vulnerability identification is crucial to protect the software systems from attacks for cyber security. It is especially important to localize the vulnerable functions among the source code to facilitate the fix. However, it is a challenging and tedious process, and also requires specialized security expertise. Inspired by the work on manually-defined patterns of vulnerabilities from various code representation graphs and the recent advance on graph neural networks, we propose Devign, a general graph neural network based model for graph-level classification through learning on a rich set of code semantic representations. It includes a novel Conv module to efficiently extract useful features in the learned rich node representations for graph-level classification. The model is trained over manually labeled datasets built on 4 diversified large-scale open-source C projects that incorporate high complexity and variety of real source code instead of synthesis code used in previous works. The results of the extensive evaluation on the datasets demonstrate that Devign outperforms the state of the arts significantly with an average of 10.51% higher accuracy and 8.68\% F1 score, increases averagely 4.66% accuracy and 6.37% F1 by the Conv module.

Details of downstream tasks datasets.

The main goal of DISCO is to generate functionality-aware code embeddings, producing similar representations for code clones and differentiating security bugs from the benign programs.
Our data is collected from either the open-source projects, respecting corresponding licences' restrictions, or publicly available benchmarks. Meanwhile, throughout the paper we make sure
to summarize the paper's main claims. We also discussed DISCO's limitation and potential future work for clone detection in Section REF . We report our model configurations and experiment details in Appendix REF .

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021.
Unified
pre-training for program understanding and generation.
In Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, pages 2655–2668, Online. Association for Computational
Linguistics.
Qurat Ul Ain, Wasi Haider Butt, Muhammad Waseem Anwar, Farooque Azam, and Bilal
Maqbool. 2019.
A systematic
review on code clone detection.
IEEE Access, 7:86121–86144.
Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018.
Learning to
represent programs with graphs.
In International Conference on Learning Representations.
Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. 2021.
Self-supervised bug detection and repair.
In NeurIPS.
Nghi D. Q. Bui, Yijun Yu, and Lingxiao Jiang. 2021.
Self-supervised
contrastive learning for code retrieval and summarization via
semantic-preserving transformations.
In SIGIR '21, page 511–521.
Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCarley, Yunhui Zheng,
Gaetano Rossiello, Alessandro Morari, Jim Laredo, Veronika Thost, Yufan
Zhuang, and Giacomo Domeniconi. 2020.
Exploring software
naturalness through neural language models.
S. Chakraborty, Y. Ding, M. Allamanis, and B. Ray. 2020.
Codit: Code editing
with tree-based neural models.
IEEE Transactions on Software Engineering, pages 1–1.
Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2021.
Deep learning based
vulnerability detection: Are we there yet.
IEEE Transactions on Software Engineering, pages 1–1.
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer.
2002.
Smote: Synthetic minority over-sampling technique.
J. Artif. Int. Res., 16(1):321–357.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.
A simple
framework for contrastive learning of visual representations.
In Proceedings of the 37th International Conference on Machine
Learning, pages 1597–1607. PMLR.
CVE-2019-12730. 2019.
https://nvd.nist.gov/vuln/detail/CVE-2019-12730.
CVE-2020-24020. 2020.
https://nvd.nist.gov/vuln/detail/CVE-2020-24020.
CVE-2021-3449. 2021.
https://nvd.nist.gov/vuln/detail/CVE-2021-3449.
CVE-2021-38094. 2021.
https://nvd.nist.gov/vuln/detail/CVE-2021-38094.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
BERT: Pre-training of
deep bidirectional transformers for language understanding.
In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1, pages 4171–4186, Minneapolis, Minnesota.
Association for Computational Linguistics.
Yangruibo Ding, Baishakhi Ray, Devanbu Premkumar, and Vincent J. Hellendoorn.
2020.
Patching as
translation: the data and the metaphor.
In 35th IEEE/ACM International Conference on Automated Software
Engineering, ASE '20.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun
Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.
CodeBERT: A pre-trained model for programming and natural languages.
In Findings of the Association for Computational Linguistics:
EMNLP 2020, pages 1536–1547, Online. Association for Computational
Linguistics.
Marco Funaro, Daniele Braga, Alessandro Campi, and Carlo Ghezzi. 2010.
A hybrid approach (syntactic and textual) to clone detection.
In Proceedings of the 4th International Workshop on Software
Clones, pages 79–80.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.
SimCSE: Simple contrastive learning of sentence embeddings.
In Empirical Methods in Natural Language Processing (EMNLP).
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021.
DeCLUTR:
Deep contrastive learning for unsupervised textual representations.
In Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers), pages 879–895, Online.
Association for Computational Linguistics.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou,
Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng,
Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming
Zhou. 2021.
Graphcode{bert}: Pre-training code representations with data
flow.
In International Conference on Learning Representations.
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and
David Bieber. 2020.
Global relational
models of source code.
In International Conference on Learning Representations.
Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph E. Gonzalez, and
Ion Stoica. 2020.
Contrastive code representation learning.
arXiv preprint.
Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. 2021.
Treebert: A tree-based pre-trained model for programming language.
ArXiv, abs/2105.12485.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020.
Learning and evaluating contextual embedding of source code.
In ICML 2020.
Seulbae Kim, Seunghoon Woo, Heejo Lee, and Hakjoo Oh. 2017.
Vuddy: A scalable
approach for vulnerable code clone discovery.
In 2017 IEEE Symposium on Security and Privacy (SP), pages
595–614.
Taku Kudo and John Richardson. 2018.
SentencePiece: A
simple and language independent subword tokenizer and detokenizer for neural
text processing.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations, pages 66–71, Brussels,
Belgium. Association for Computational Linguistics.
Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Hanchao Qi, and Jie Hu. 2016.
Vulpecker: An
automated vulnerability detection system based on code similarity analysis.
In Proceedings of the 32nd Annual Conference on Computer
Security Applications, ACSAC '16, page 201–213.
Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen.
2018a.
Sysevr: A framework for
using deep learning to detect software vulnerabilities.
Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng,
and Yuyi Zhong. 2018b.
Vuldeepecker: A deep learning-based system for vulnerability
detection.
In Proceedings of the 25th Annual Network and Distributed
System Security Symposium (NDSS`2018).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Roberta: A robustly
optimized BERT pretraining approach.
CoRR, abs/1907.11692.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio
Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong
Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan,
Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021.
Codexglue: A machine learning benchmark dataset for code
understanding and generation.
CoRR, abs/2102.04664.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016.
Convolutional neural networks over tree structures for programming
language processing.
In Proceedings of the Thirtieth AAAI Conference on Artificial
Intelligence, pages 1287–1293.
Long N. Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James T. Anibal, Alec
Peltekian, and Yanfang Ye. 2021.
Cotext: Multi-task learning
with code-text transformer.
CoRR, abs/2105.08645.
Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka.
2021.
Contrastive
learning with hard negative samples.
In International Conference on Learning Representations.
Baptiste Rozière, Marie-Anne Lachaux, Marc Szafraniec, and Guillaume
Lample. 2021.
DOBF: A deobfuscation
pre-training objective for programming languages.
CoRR, abs/2102.07492.
Abdullah Sheneamer, Swarup Roy, and Jugal Kalita. 2018.
A detection framework for semantic code clones and obfuscated code.
Expert Systems with Applications, 97:405–420.
Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and
Mohammad Mamun Mia. 2014.
Towards a big data curated benchmark of inter-project code clones.
In 2014 IEEE International Conference on Software Maintenance
and Evolution, pages 476–480. IEEE.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.
Attention is all you need.
In Proceedings of the 31st International Conference on Neural
Information Processing Systems, NIPS'17, page 6000–6010.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021.
Codet5: Identifier-aware unified pre-trained encoder-decoder models
for code understanding and generation.
In Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing, EMNLP 2021.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020.
CLEAR: contrastive
learning for sentence representation.
CoRR, abs/2012.15466.
Fangke Ye, Shengtian Zhou, Anand Venkat, Ryan Marcus, Nesime Tatbul,
Jesmin Jahan Tithi, Paul Petersen, Timothy G. Mattson, Tim Kraska, Pradeep
Dubey, Vivek Sarkar, and Justin Gottschlich. 2020.
MISIM: an end-to-end
neural code similarity system.
CoRR, abs/2006.05265.
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019.
Devign: Effective vulnerability identification by learning
comprehensive program semantics via graph neural networks.
In Advances in Neural Information Processing Systems, pages
10197–10207.
Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and
Stephan Günnemann. 2021.
Language-agnostic representation learning of source code from
structure and context.
In International Conference on Learning Representations
(ICLR).

Appendix

Bug Injection Heuristics and Common Weakness Enumeration Types

Our automated bug injection heuristics are motivated by the real-world security bugs that are always small but hazardous. We empirically conclude the frequently happened vulnerable patterns based on the concrete CWE types. Table {{table:0a91cb1e-9ad3-4828-a63c-7a34f688fa89}}  shows that each of our operation is relating to several CWE types. We inject all these security issues automatically and ask model to distinguish them with the benign samples.

Node Type Details

We parse the source code into ASTs and extract the node type and parent node type for each token. Table {{table:138f5fa0-d424-4103-ae60-67bd22b9cd8f}}  shows an example after parsing. We can see that, with the parent node type, each token can be well embedded with its local structure contexts. Considering two tokens that are distant from each other: if and else. With only node types, we just know these two tokens are keywords, but with parent node type, we can easily know that they are from the same if-statement and they are siblings in the AST.

Dataset

Pre-training We collect our dataset from C and Java Github repositories. Our main dataset is the combination of Java small and C small. From Table {{table:fe2539cd-bd1f-4ed0-8840-ee7ab0c62084}} , we can see that our dataset is significantly smaller than the existing pre-trained models. For an ablation study (§ REF ) with enlarged dataset, we collect a medium dataset of C language. We have seen the improvement using such larger dataset, but even medium dataset is still much smaller than other datasets.
Datasets for downstream tasks We provide dataset details of our downstream tasks in Table {{table:5115930b-4d8b-4e05-9d38-b8c971672ec6}} . Noted that for POJ-104 {{cite:8f61658}}, Table {{table:5115930b-4d8b-4e05-9d38-b8c971672ec6}}  only shows the number of code samples, and we follow the design of CodeXGLUE that build positive and negative pairs during the minibatch generation. The amount of pairs for training is much larger than the number of samples.
Comparison of pre-training dataset size between ours and other related work

Dataset
Instance Count
Total Size
DISCO

Java small
187 K
992 MB
C small
64 K
865 MB
C medium
860 K
12 GB
CodeBERT
8.5 M
20 GB
GraphCodeBERT
2.3 M
20 GB
CuBERT
7.4 M
-
DOBF
-
45 GB
PLBART
-
576 GB
{{table:eead0cd8-948b-4584-9f39-aa68f387a059}}{{table:fe2539cd-bd1f-4ed0-8840-ee7ab0c62084}}Common Weakness Enumeration (CWE) types covered by our bug injection heuristic

Operation
Potential CWE types
6*Misuse of Data Type
CWE-190: Integer overflow

CWE-191: Integer Underflow

CWE-680: Integer Overflow to Buffer Overflow

CWE-686: Function Call With Incorrect Argument Type

CWE-704: Incorrect Type Conversion or Cast

CWE-843: Access of Resource Using Incompatible Type
3*Misuse of Pointer
CWE-476: NULL Pointer Dereference

CWE-824: Access of Uninitialized Pointer

CWE-825: Expired Pointer Dereference
10*Change of Conditional Statements
CWE-120: Buffer Overflow

CWE-121: Stack-based Buffer overflow

CWE-122: Heap-based Buffer overflow

CWE-124: Buffer Underflow

CWE-125: Out-of-bounds Read

CWE-126: Buffer Over-read

CWE-129: Improper Validation of Array Index

CWE-787: Out-of-bounds Write

CWE-788: Access of Memory Location After End of Buffer

CWE-823: Use of Out-of-range Pointer Offset
4*Misuse of Values
CWE-369: Divide By Zero

CWE-456: Missing Initialization of a Variable

CWE-457: Use of Uninitialized Variable

CWE-908: Use of Uninitialized Resource
5*Change of Function Calls
CWE-683: Function Call With Incorrect Order of Arguments

CWE-685: Function Call With Incorrect Number of Arguments

CWE-686: Function Call With Incorrect Argument Type

CWE-687: Function Call With Incorrectly Specified Argument Value

CWE-688: Function Call With Incorrect Variable or Reference
{{table:616bbc98-d9dd-4e8a-b67b-ab74cea4970d}}{{table:0a91cb1e-9ad3-4828-a63c-7a34f688fa89}}Examples for tokens and their AST node types

token
node type
parent node type
token
node type
parent node type
int
type
func_definition
)
punctuation
parenthesized_expr
foo
identifier
func_declarator
return
keyword
return_stmt
(
punctuation
param_list
(
punctuation
parenthesized_expr
int
type
parameter_declaration
1
number_literal
parenthesized_expr
bar
identifier
parameter_declaration
)
punctuation
parenthesized_expr
)
punctuation
parameter_list
;
punctuation
return_stmt
{
punctuation
compount_stmt
else
keyword
if_stmt
if
keyword
if_stmt
return
keyword
return_stmt
(
punctuation
parenthesized_expr
(
punctuation
parenthesized_expr
bar
identifier
binary_expr
0
number_literal
parenthesized_expr
{{formula:3a40cbf5-b477-4c61-a456-44fb637f2b8e}}
operator
binary_expr
)
number_literal
parenthesized_expr
5
number_literal
binary_expr
;
punctuation
return_stmt

}
punctuation
compount_stmt
{{table:c7a1f038-f9ec-4b53-8297-391bf80faf5a}}{{table:138f5fa0-d424-4103-ae60-67bd22b9cd8f}}Details of downstream tasks datasets.

Task
Dataset
Language
Train
Valid
Test
2*Vulnerability Detection
{{cite:0bb8702}}
C/C++
15,867
2,268
4,535

{{cite:9badd66}}
C/C++
21,854
2,732
2,732
2*Clone Detection
{{cite:8f61658}}
C/C++
32,000
8,000
12,000

{{cite:5daec6a}}
Java
901,028
415,416
415,416
{{table:21d6dfc5-974e-49ad-9267-1504052d25fc}}{{table:5115930b-4d8b-4e05-9d38-b8c971672ec6}}An example in REVEAL dataset. The patched code happens to be in the train split and the buggy code is in the test split. During inference, DISCO MLM+CLR{{formula:bd9da025-bd12-4502-81f6-d169d48f5f03}}  model can correctly predict the buggy code as vulnerable, while MLM+CLR{{formula:98572585-2b9f-44f3-bdb3-b6801ebb5096}} predicts it as benign.
{{figure:6a2aaee9-b100-4172-b0da-cc679e90d80e}}

Configuration

DISCO is built based on a stack of 12 layers transformer encoder with 12 attention heads and 768 hidden sizes. The model is implemented with PyTorch-1.9.0 and Huggingface-transformer-4.12.3 https://github.com/huggingface/transformers/tree/c439752482759c94784e11a87dcbf08ce69dccf3.
Longer sequences are disproportionately expensive so we follow the original BERT design by pre-training the model with short sequence length for first 70% steps and long sequence length for the rest 30% steps to learn the positional embeddings. DISCO is trained with Java small and C small for 24 hours in total with two 32GB NVIDIA Tesla V100 GPUs, using batch size of 128 with max sequence length of 256 tokens and batch size of 64 sequences with max sequence length 512 tokens.
DISCO is also trained with C medium for 3 days, using batch size of 1024 sequences with max sequence length of 256 tokens and batch size of 512 sequences and max sequence length of 512 tokens.
We use the Adam optimizer and 1e-4 as the pre-training learning rate. For fine-tuning tasks, we use batch size of 8 and the learning of 8e-6. We train the model with train split and evaluate the model during the training using validation split. We pick the model with best validation performance for testing.

Case Study

We studied the model performance on REVEAL dataset for vulnerability detection. Figure {{figure:6a2aaee9-b100-4172-b0da-cc679e90d80e}}  shows two samples inside REVEAL. We can recognize that they are from the same program. We further checked the details of these two example and we found the code on the left is a buggy version, and it is fixed by adding an argument of value 0 to the function call. This real-world situation actually matches our "Change of Function Calls" (§ REF ) bug injection operation. In the REVEAL dataset, the patched code is in the training corpus while the buggy one is in the test split. Interestingly, during the inference, DISCO MLM+CLR{{formula:616375ca-013e-42fd-9992-d417a4f41577}}  can correctly predict the buggyiess while MLM+CLR{{formula:1ce6324c-ea6b-413b-a183-3d09410c3935}} fails. This empirically prove our bug injected samples can help the model identify small but siginicant real-world vulnerabilities.


the first
contextualized word representation

a longer-trained BERT variant using more data

a BERT variant with shared weights and a factorized word embedding

a generalized autoregressive pre-training using bidirectional contexts

Deep Contextualized Word Representations

Abstract: We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Abstract: Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at this https URL.

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Abstract: With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

Representative pre-trained language models in the general domain. NSP means the next sentence prediction task.

Interestingly, the computer vision {{cite:c930009}} and computational biology communities also borrow some insights to design their models, see ViT {{cite:c930009}} for vision and AlphaFold2 {{cite:22a2643}} for protein.
In Table {{table:b4310964-4a7c-4fc4-b37c-31f5e3456371}} , we introduce some typical pre-trained language models in general NLP domains, based on these two backbone neural networks.


Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers

Abstract: Although various techniques have been proposed to generate adversarial samples for white-box attacks on text, little attention has been paid to a black-box attack, which is a more realistic scenario. In this paper, we present a novel algorithm, DeepWordBug, to effectively generate small text perturbations in a black-box setting that forces a deep-learning classifier to misclassify a text input. We develop novel scoring strategies to find the most important words to modify such that the deep classifier makes a wrong prediction. Simple character-level transformations are applied to the highest-ranked words in order to minimize the edit distance of the perturbation. We evaluated DeepWordBug on two real-world text datasets: Enron spam emails and IMDB movie reviews. Our experimental results indicate that DeepWordBug can reduce the classification accuracy from 99% to 40% on Enron and from 87% to 26% on IMDB. Our results strongly demonstrate that the generated adversarial sequences from a deep-learning model can similarly evade other deep models.

TextBugger: Generating Adversarial Text Against Real-world Applications

Abstract: Deep Learning-based Text Understanding (DLTU) is the backbone technique behind various applications, including question answering, machine translation, and text classification. Despite its tremendous popularity, the security vulnerabilities of DLTU are still largely unknown, which is highly concerning given its increasing use in security-sensitive applications such as sentiment analysis and toxic content detection. In this paper, we show that DLTU is inherently vulnerable to adversarial text attacks, in which maliciously crafted texts trigger target DLTU systems and services to misbehave. Specifically, we present TextBugger, a general attack framework for generating adversarial texts. In contrast to prior works, TextBugger differs in significant ways: (i) effective -- it outperforms state-of-the-art attacks in terms of attack success rate; (ii) evasive -- it preserves the utility of benign text, with 94.9\% of the adversarial text correctly recognized by human readers; and (iii) efficient -- it generates adversarial text with computational complexity sub-linear to the text length. We empirically evaluate TextBugger on a set of real-world DLTU systems and services used for sentiment analysis and toxic content detection, demonstrating its effectiveness, evasiveness, and efficiency. For instance, TextBugger achieves 100\% success rate on the IMDB dataset based on Amazon AWS Comprehend within 4.61 seconds and preserves 97\% semantic similarity. We further discuss possible defense mechanisms to mitigate such attack and the adversary's potential countermeasures, which leads to promising directions for further research.

Synthetic and Natural Noise Both Break Neural Machine Translation

Abstract: Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems. Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise.

Character modifications of the main character-level adversarial attacks. Y: Yes, N: No.

Character level information has been widely used in NLP for text classification tasks. {{cite:b5ea16a}} presents a pioneering work on applying convolutional neural networks (CNN) for text classification at the character level. This work has paved the way for character-level operations. In Table {{table:3e3e9b77-548e-4dba-a3ed-09dc2d54b733}} , we summarize several character-level perturbations. These primarily include insertion, deletion, substitution/replacing, swapping, and/or scrambling a character. In Table {{table:a9334411-5602-4f57-991b-32677c5d94fb}} , we list sample works on character-level attacks with their model accessibility, attack type, targeted model, application, or task.


PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts

Abstract: We present PubMed 200k RCT, a new dataset based on PubMed for sequential sentence classification. The dataset consists of approximately 200,000 abstracts of randomized controlled trials, totaling 2.3 million sentences. Each sentence of each abstract is labeled with their role in the abstract using one of the following classes: background, objective, method, result, or conclusion. The purpose of releasing this dataset is twofold. First, the majority of datasets for sequential short-text classification (i.e., classification of short texts that appear in sequences) are small: we hope that releasing a new large dataset will help develop more accurate algorithms for this task. Second, from an application perspective, researchers need better tools to efficiently skim through the literature. Automatically classifying each sentence in an abstract would help researchers read abstracts more efficiently, especially in fields where abstracts may be long, such as the medical field.

Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction

Abstract: We introduce a multi-task setup of identifying and classifying entities, relations, and coreference clusters in scientific articles. We create SciERC, a dataset that includes annotations for all three tasks and develop a unified framework called Scientific Information Extractor (SciIE) for with shared span representations. The multi-task setup reduces cascading errors between tasks and leverages cross-sentence relations through coreference links. Experiments show that our multi-task model outperforms previous models in scientific information extraction without using any domain-specific features. We further show that the framework supports construction of a scientific knowledge graph, which we use to analyze information in scientific literature.

The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures

Abstract: Materials science literature contains millions of materials synthesis procedures described in unstructured natural language text. Large-scale analysis of these synthesis procedures would facilitate deeper scientific understanding of materials synthesis and enable automated synthesis planning. Such analysis requires extracting structured representations of synthesis procedures from the raw text as a first step. To facilitate the training and evaluation of synthesis extraction models, we introduce a dataset of 230 synthesis procedures annotated by domain experts with labeled graphs that express the semantics of the synthesis sentences. The nodes in this graph are synthesis operations and their typed arguments, and labeled edges specify relations between the nodes. We describe this new resource in detail and highlight some specific challenges to annotating scientific text with shallow semantic structure. We make the corpus available to the community to promote further research and development of scientific information extraction systems.

SemEval 2017 Task 10: ScienceIE - Extracting Keyphrases and Relations from Scientific Publications

Abstract: We describe the SemEval task of extracting keyphrases and relations between them from scientific documents, which is crucial for understanding which publications describe which processes, tasks and materials. Although this was a new task, we had a total of 26 submissions across 3 evaluation scenarios. We expect the task and the findings reported in this paper to be relevant for researchers working on understanding scientific content, as well as the broader knowledge base population and information extraction communities.

Summary of downstream datasets relevant to each domain in the research paper stream.

This paper stream consists of the full text of research papers published in four research areas: biomedical, computer science, material science, and physics, filtered from the S2ORC datasetWe use the 20200705v1 version of the S2ORC dataset at https://github.com/allenai/s2orc, which are presented sequentially to the model. For each domain, we evaluate downstream performance over two datasets. The downstream tasks span over various tasks such as relation extraction and named entity recognition, and are summarized in Table {{table:66376fa0-a019-4d93-88ba-937848cd4fef}} . We detail these datasets in Appendix .


PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Abstract: Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds and well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Abstract: Few prior works study deep learning on point sets. PointNet by Qi et al. is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds.

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

Abstract: In many robotics and VR/AR applications, 3D-videos are readily-available sources of input (a continuous sequence of depth images, or LIDAR scans). However, those 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose the generalized sparse convolution that encompasses all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in the 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that convolutional neural networks with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise, outperform 3D convolutional neural networks and are faster than the 3D counterpart in some cases.

Evaluation Benchmarks

Challenge: High Input Sparsity.
Unlike the input of image CNNs which are dense tensors, point cloud is naturally sparse. Figure {{figure:47d4a024-bf8f-4958-9748-32e5298bb958}}  (left) plots the input sparsity of five mainstream point cloud datasets ModelNet40 {{cite:643e3af}}, ShapeNet {{cite:b6907bc}}, KITTI {{cite:5098937}}, S3DIS {{cite:889b774}} and SemanticKITTI {{cite:0127c94}}, with details summarized in Table {{table:2e8e1514-d946-47d6-9768-61d10a456a6f}} . Point clouds of 3D indoor scenes and objects have a density of {{formula:37250a21-1b86-4512-b17b-93cfe1d763ef}} , and the 3D outdoor scenes are even sparser, reaching a density of less than {{formula:6968e011-2cf2-4989-a480-2a054454b10a}} . In contrast, ImageNet has 100% density at the input and 50% density on average after ReLU, which is up to four orders of magnitude denser than point cloud.

Benchmarks.
We pick 8 diverse point cloud network benchmarks across assorted application domains, including object classification, part segmentation, detection, and semantic segmentation. As shown in Table {{table:2e8e1514-d946-47d6-9768-61d10a456a6f}} , the networks include both classical and state-of-the-art ones and cover all categories in Table {{table:52721c3e-570f-4f6b-aa7a-7c1390425dd4}} . The selected datasets also contain various sizes and modalities for input point clouds, from daily objects to indoor scenes to spacious outdoor scenes. Such extensive benchmarks allow us to evaluate PointAcc thoroughly.


Topic Modeling in Embedding Spaces

Abstract: Topic modeling analyzes documents to learn meaningful patterns of words. However, existing topic models fail to learn interpretable topics when working with large and heavy-tailed vocabularies. To this end, we develop the Embedded Topic Model (ETM), a generative model of documents that marries traditional topic models with word embeddings. In particular, it models each word with a categorical distribution whose natural parameter is the inner product between a word embedding and an embedding of its assigned topic. To fit the ETM, we develop an efficient amortized variational inference algorithm. The ETM discovers interpretable topics even with large vocabularies that include rare words and stop words. It outperforms existing document models, such as latent Dirichlet allocation (LDA), in terms of both topic quality and predictive performance.

The Poisson Gamma Belief Network

Abstract: To infer a multilayer representation of high-dimensional count vectors, we propose the Poisson gamma belief network (PGBN) that factorizes each of its layers into the product of a connection weight matrix and the nonnegative real hidden units of the next layer. The PGBN's hidden layers are jointly trained with an upward-downward Gibbs sampler, each iteration of which upward samples Dirichlet distributed connection weight vectors starting from the first layer (bottom data layer), and then downward samples gamma distributed hidden units starting from the top hidden layer. The gamma-negative binomial process combined with a layer-wise training strategy allows the PGBN to infer the width of each layer given a fixed budget on the width of the first layer. The PGBN with a single hidden layer reduces to Poisson factor analysis. Example results on text analysis illustrate interesting relationships between the width of the first layer and the inferred network structure, and demonstrate that the PGBN, whose hidden units are imposed with correlated gamma priors, can add more layers to increase its performance gains over Poisson factor analysis, given the same limit on the width of the first layer.

The Poisson Gamma Belief Network

The Poisson Gamma Belief Network

Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC

Abstract: It is challenging to develop stochastic gradient based scalable inference for deep discrete latent variable models (LVMs), due to the difficulties in not only computing the gradients, but also adapting the step sizes to different latent factors and hidden layers. For the Poisson gamma belief network (PGBN), a recently proposed deep discrete LVM, we derive an alternative representation that is referred to as deep latent Dirichlet allocation (DLDA). Exploiting data augmentation and marginalization techniques, we derive a block-diagonal Fisher information matrix and its inverse for the simplex-constrained global model parameters of DLDA. Exploiting that Fisher information matrix with stochastic gradient MCMC, we present topic-layer-adaptive stochastic gradient Riemannian (TLASGR) MCMC that jointly learns simplex-constrained global parameters across all layers and topics, with topic and layer specific learning rates. State-of-the-art results are demonstrated on big data sets.

Comparison of per-heldout-word perplexity on three different datasets.

where {{formula:93768908-c6ce-4643-a7ef-fbe912e93cbf}}  is the total number of collected samples and {{formula:7d3c171d-1584-4f38-85d3-c63e2956bf7b}} .
As shown in Tab. {{table:73087212-5a16-4008-98a4-743d007bd1f2}} , overall, DLDA-Gibbs perform best in terms of PPL, which is not surprising as they can sample from the true posteriors given a sufficiently large number of Gibbs sampling iterations. DLDA-TLASGR is a mini-batch based algorithm that is much more scalable in training than DLDA-Gibbs, at the expense of slightly degraded performance in out-of-sample prediction. Apart from these deep probabilistic topic models, Gaussian SawETM outperforms other neural topic models in all datasets. In particular, compared with Gaussian based single layer neural topic model, WHAI performs better, which can be attributed to two aspects, the first is the effectiveness of Weibull distribution in modeling sparse and non-negative document latent representations, and the second is aggregating the information in both the higher (prior) layer and that upward propagated to the current layer via inference network. Benefiting from the embedding decoder and the Sawtooth Connection that builds the dependency of hierarchical topics, SawETM performs better than WHAI, especially at deeper layers. However, modeling topics with point vectors, SawETM cannot capture the uncertainties of topics. Gaussian SawETM represents the topics with Gaussian-distributed embeddings to capture the uncertainties, and achieves promising improvements over SawETM, especially for modeling complex and big corpora such as Wiki. Note that although the methods in DLDA group get better performance, they require iteratively sampling to infer latent document representations in the test stage, which limits their application {{cite:f57857b}}. By contrast, Gaussian SawETM can infer latent representations via direct projection, which makes it both scalable to large corpora and fast in out-of-sample prediction.


Deep Learning Based Vulnerability Detection: Are We There Yet?

Abstract: Automated detection of software vulnerabilities is a fundamental problem in software security. Existing program analysis techniques either suffer from high false positives or false negatives. Recent progress in Deep Learning (DL) has resulted in a surge of interest in applying DL for automated vulnerability detection. Several recent studies have demonstrated promising results achieving an accuracy of up to 95 percent at detecting vulnerabilities. In this paper, we ask, “how well do the state-of-the-art DL-based techniques perform in a real-world vulnerability prediction scenario?” To our surprise, we find that their performance drops by more than 50 percent. A systematic investigation of what causes such precipitous performance drop reveals that existing DL-based vulnerability prediction approaches suffer from challenges with the training data (e.g., data duplication, unrealistic distribution of vulnerable classes, etc.) and with the model choices (e.g., simple token-based models). As a result, these approaches often do not learn features related to the actual cause of the vulnerabilities. Instead, they learn unrelated artifacts from the dataset (e.g., specific variable/function names, etc.). Leveraging these empirical findings, we demonstrate how a more principled approach to data collection and model design, based on realistic settings of vulnerability prediction, can lead to better solutions. The resulting tools perform significantly better than the studied baseline—up to 33.57 percent boost in precision and 128.38 percent boost in recall compared to the best performing model in the literature. Overall, this paper elucidates existing DL-based vulnerability prediction systems’ potential issues and draws a roadmap for future DL-based vulnerability prediction research.

Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks

Abstract: Vulnerability identification is crucial to protect the software systems from attacks for cyber security. It is especially important to localize the vulnerable functions among the source code to facilitate the fix. However, it is a challenging and tedious process, and also requires specialized security expertise. Inspired by the work on manually-defined patterns of vulnerabilities from various code representation graphs and the recent advance on graph neural networks, we propose Devign, a general graph neural network based model for graph-level classification through learning on a rich set of code semantic representations. It includes a novel Conv module to efficiently extract useful features in the learned rich node representations for graph-level classification. The model is trained over manually labeled datasets built on 4 diversified large-scale open-source C projects that incorporate high complexity and variety of real source code instead of synthesis code used in previous works. The results of the extensive evaluation on the datasets demonstrate that Devign outperforms the state of the arts significantly with an average of 10.51% higher accuracy and 8.68\% F1 score, increases averagely 4.66% accuracy and 6.37% F1 by the Conv module.

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Abstract:
Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas.
To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task.
We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.

Details of downstream tasks datasets.

Datasets for downstream tasks We provide dataset details of our downstream tasks in Table {{table:201999ed-b55a-443c-98a9-0d75a0193a16}} . Noted that for POJ-104 {{cite:1fc699a}}, Table {{table:201999ed-b55a-443c-98a9-0d75a0193a16}}  only shows the number of code samples, and we follow the design of CodeXGLUE that build positive and negative pairs during the minibatch generation. The amount of pairs for training is much larger than the number of samples.


a generalized autoregressive pretraining using bidirectional contexts

Deep contextualized word representations

Abstract: We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Abstract: Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at this https URL.

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Abstract: With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

The backbone of most pre-trained language models (e.g., BERT and GPT) is a neural network called `Transformer ' building upon self-attention networks (SANs) and feed-forward networks (FFNs). SAN is used to facilitate interaction between tokens, while FNN is used to refine the token presentation using non-linear transformation.
Since Transformer has been the de facto backbone to replace recurrent and convolutional units, BERT and GPT adopt the Transformer as the backbone network. The transformer is superior in terms of capacity and scalability thanks to, 1) discard recurrent units and process tokens more efficiently in parallel with the position embeddings{{cite:61e3db8}}, {{cite:cedf33b}}, 2) relieve saturation issue of expressive power with large-scale data and very deep layers due to the well-designed architecture including residual connections, layer normalization, and etc.
Interestingly, AlphaFold2 {{cite:f5c40a0}} also borrows some insights to design the so-called `Evoformer' as the core component in its architecture. In Table {{table:38e1a654-6b9f-419e-9317-c137a29c351f}} , we introduce some typical pre-trained language models in general NLP domains, based on these two backbone neural networks.


MedSTS: A Resource for Clinical Semantic Textual Similarity

Abstract: The wide adoption of electronic health records (EHRs) has enabled a wide range of applications leveraging EHR data. However, the meaningful use of EHR data largely depends on our ability to efficiently extract and consolidate information embedded in clinical text where natural language processing (NLP) techniques are essential. Semantic textual similarity (STS) that measures the semantic similarity between text snippets plays a significant role in many NLP applications. In the general NLP domain, STS shared tasks have made available a huge collection of text snippet pairs with manual annotations in various domains. In the clinical domain, STS can enable us to detect and eliminate redundant information that may lead to a reduction in cognitive burden and an improvement in the clinical decision-making process. This paper elaborates our efforts to assemble a resource for STS in the medical domain, MedSTS. It consists of a total of 174,629 sentence pairs gathered from a clinical corpus at Mayo Clinic. A subset of MedSTS (MedSTS_ann) containing 1,068 sentence pairs was annotated by two medical experts with semantic similarity scores of 0-5 (low to high similarity). We further analyzed the medical concepts in the MedSTS corpus, and tested four STS systems on the MedSTS_ann corpus. In the future, we will organize a shared task by releasing the MedSTS_ann corpus to motivate the community to tackle the real world clinical problems.

The 2019 n2c2/OHNLP Track on Clinical Semantic Textual Similarity: Overview

Abstract: Background Semantic textual similarity is a common task in the general English domain to assess the degree to which the underlying semantics of 2 text segments are equivalent to each other. Clinical Semantic Textual Similarity (ClinicalSTS) is the semantic textual similarity task in the clinical domain that attempts to measure the degree of semantic equivalence between 2 snippets of clinical text. Due to the frequent use of templates in the Electronic Health Record system, a large amount of redundant text exists in clinical notes, making ClinicalSTS crucial for the secondary use of clinical text in downstream clinical natural language processing applications, such as clinical text summarization, clinical semantics extraction, and clinical information retrieval. Objective Our objective was to release ClinicalSTS data sets and to motivate natural language processing and biomedical informatics communities to tackle semantic text similarity tasks in the clinical domain. Methods We organized the first BioCreative/OHNLP ClinicalSTS shared task in 2018 by making available a real-world ClinicalSTS data set. We continued the shared task in 2019 in collaboration with National NLP Clinical Challenges (n2c2) and the Open Health Natural Language Processing (OHNLP) consortium and organized the 2019 n2c2/OHNLP ClinicalSTS track. We released a larger ClinicalSTS data set comprising 1642 clinical sentence pairs, including 1068 pairs from the 2018 shared task and 1006 new pairs from 2 electronic health record systems, GE and Epic. We released 80% (1642/2054) of the data to participating teams to develop and fine-tune the semantic textual similarity systems and used the remaining 20% (412/2054) as blind testing to evaluate their systems. The workshop was held in conjunction with the American Medical Informatics Association 2019 Annual Symposium. Results Of the 78 international teams that signed on to the n2c2/OHNLP ClinicalSTS shared task, 33 produced a total of 87 valid system submissions. The top 3 systems were generated by IBM Research, the National Center for Biotechnology Information, and the University of Florida, with Pearson correlations of r=.9010, r=.8967, and r=.8864, respectively. Most top-performing systems used state-of-the-art neural language models, such as BERT and XLNet, and state-of-the-art training schemas in deep learning, such as pretraining and fine-tuning schema, and multitask learning. Overall, the participating systems performed better on the Epic sentence pairs than on the GE sentence pairs, despite a much larger portion of the training data being GE sentence pairs. Conclusions The 2019 n2c2/OHNLP ClinicalSTS shared task focused on computing semantic similarity for clinical text sentences generated from clinical notes in the real world. It attracted a large number of international teams. The ClinicalSTS shared task could continue to serve as a venue for researchers in natural language processing and medical informatics communities to develop and improve semantic textual similarity techniques for clinical text.

Datasets used in the biomedical sentence similarity task.

Existing pre-trained language models in the biomedical domain and general domain such as BERT, RoBERTa, BioBERT, SciBERT, ClinicalBERT, BlueBERT, PubMedBERT, and BioMegatron have achieved great performance in the task, in which the BioMegatron yielded the best performance due to domain-specific pre-training and task-specific fine-tuning. However, these models are pre-trained for yielding embeddings in the token level with the contextual information. To better capture the semantic information at the sentence level, Chen et al {{cite:5aec93d}} proposed the first pre-trained open set sentence embeddings in the biomedical domain, called BioSentVec, which is trained on over 30 million documents from both biomedical literatures such as PubMed and clinical notes such as the MIMIC- III Clinical Database. Compared with existing word embeddings and sentence encoder-based methods, it yields better performance on both sentence similarity and text classification tasks, due to better capture the sentence semantic. In {{cite:d36f6f8}}, they empirically compared the performance of traditional deep learning methods such as random forest, RNN, CNN with the pre-trained models BERT and BioBERT, which shown that pre-trained language models are more effective in capture the sentence semantic due to pre-trained in large scale of corpora. Moreover, in {{cite:9afc619}}, they further employed the pre-trained sentence embedding BioSentVec to improve the traditional deep learning models: random forest and the encoder network on finding similar sentences. Yang et al {{cite:59a8387}} explored three pre-trained models including BERT, XLNet, and RoBERTa for the clinical semantic textual similarity task, in which the XLNet achieves the best performance among three models. Li et al {{cite:68fc5ed}} proposed to integrate the BERT models and bidirectional recurrent neural network (Bi-RNN) to capture both contextual semantic and semantic textual similarity. The commonly used sentence similarity datasets are shown in Table {{table:44cb0ec1-941d-4a3a-9a2e-42ea4571ed65}} .


PubMedQA: A Dataset for Biomedical Research Question Answering

Abstract: We introduce PubMedQA, a novel biomedical question answering (QA) dataset collected from PubMed abstracts. The task of PubMedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?) using the corresponding abstracts. PubMedQA has 1k expert-annotated, 61.2k unlabeled and 211.3k artificially generated QA instances. Each PubMedQA instance is composed of (1) a question which is either an existing research article title or derived from one, (2) a context which is the corresponding abstract without its conclusion, (3) a long answer, which is the conclusion of the abstract and, presumably, answers the research question, and (4) a yes/no/maybe answer which summarizes the conclusion. PubMedQA is the first QA dataset where reasoning over biomedical research texts, especially their quantitative contents, is required to answer the questions. Our best performing model, multi-phase fine-tuning of BioBERT with long answer bag-of-word statistics as additional supervision, achieves 68.1% accuracy, compared to single human performance of 78.0% accuracy and majority-baseline of 55.2% accuracy, leaving much room for improvement. PubMedQA is publicly available at https://pubmedqa.github.io.

Results of the seventh edition of the BioASQ Challenge

Abstract: The results of the seventh edition of the BioASQ challenge are presented in this paper. The aim of the BioASQ challenge is the promotion of systems and methodologies through the organization of a challenge on the tasks of large-scale biomedical semantic indexing and question answering. In total, 30 teams with more than 100 systems participated in the challenge this year. As in previous years, the best systems were able to outperform the strong baselines. This suggests that state-of-the-art systems are continuously improving, pushing the frontier of research.

CliCR: a Dataset of Clinical Case Reports for Machine Reading Comprehension

Abstract: We present a new dataset for machine comprehension in the medical domain. Our dataset uses clinical case reports with around 100,000 gap-filling queries about these cases. We apply several baselines and state-of-the-art neural readers to the dataset, and observe a considerable gap in performance (20% F1) between the best human and machine readers. We analyze the skills required for successful answering and show how reader performance varies depending on the applicable skills. We find that inferences using domain knowledge and object tracking are the most frequently required skills, and that recognizing omitted information and spatio-temporal reasoning are the most difficult for the machines.

emrQA: A Large Corpus for Question Answering on Electronic Medical Records

Abstract: We propose a novel methodology to generate domain-specific large-scale question answering (QA) datasets by re-purposing existing annotations for other NLP tasks. We demonstrate an instance of this methodology in generating a large-scale QA dataset for electronic medical records by leveraging existing expert annotations on clinical notes for various NLP tasks from the community shared i2b2 datasets. The resulting corpus (emrQA) has 1 million questions-logical form and 400,000+ question-answer evidence pairs. We characterize the dataset and explore its learning potential by training baseline models for question to logical form and question to answer mapping.

Rapidly Bootstrapping a Question Answering Dataset for COVID-19

Abstract: We present CovidQA, the beginnings of a question answering dataset specifically designed for COVID-19, built by hand from knowledge gathered from Kaggle's COVID-19 Open Research Dataset Challenge. To our knowledge, this is the first publicly available resource of its type, and intended as a stopgap measure for guiding research until more substantial evaluation resources become available. While this dataset, comprising 124 question-article pairs as of the present version 0.1 release, does not have sufficient examples for supervised machine learning, we believe that it can be helpful for evaluating the zero-shot or transfer capabilities of existing models on topics specifically related to COVID-19. This paper describes our methodology for constructing the dataset and presents the effectiveness of a number of baselines, including term-based techniques and various transformer-based models. The dataset is available at http://covidqa.ai/

Datasets used in the biomedical question answering task.

Besides methods for biomedical literature corpora, other works have been devoted to proposing question answering models to acquire knowledge from unstructured electronic health records (EHR).
{{cite:7c7b6d9}} investigated the performance of various pre-trained language models including BERT, BioBERT, ClinicalBERT and XLNet on the clinical question answering. They explored the fine-tuning methods with different datasets including datasets in the general domain, biomedical and clinical corpora.
Mairittha {{cite:2536726}} explored fine-tune BERT to construct the personalized EHR question answering system. The commonly used datasets in BioQA are summarized in the Table {{table:ebc9eb60-38fd-4501-a09e-c478d230d4c9}} .