Sweedish text corpus (Wikipedia, News, Social media and Legal)

Polish text corpus (Wiki, OSCAR, Subtitles and Books)

PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data

Abstract: In natural language processing (NLP), there is a need for more resources in Portuguese, since much of the data used in the state-of-the-art research is in other languages. In this paper, we pretrain a T5 model on the BrWac corpus, an extensive collection of web pages in Portuguese, and evaluate its performance against other Portuguese pretrained models and multilingual models on three different tasks. We show that our Portuguese pretrained models have significantly better performance over the original T5 models. Moreover, we demonstrate the positive impact of using a Portuguese vocabulary. Our code and models are available at https://github.com/unicamp-dl/PTT5.

Playing with Words at the National Library of Sweden -- Making a Swedish BERT

Abstract: This paper introduces the Swedish BERT ("KB-BERT") developed by the KBLab for data-driven research at the National Library of Sweden (KB). Building on recent efforts to create transformer-based BERT models for languages other than English, we explain how we used KB's collections to create and train a new language-specific BERT model for Swedish. We also present the results of our model in comparison with existing models - chiefly that produced by the Swedish Public Employment Service, Arbetsf\"ormedlingen, and Google's multilingual M-BERT - where we demonstrate that KB-BERT outperforms these in a range of NLP tasks from named entity recognition (NER) to part-of-speech tagging (POS). Our discussion highlights the difficulties that continue to exist given the lack of training data and testbeds for smaller languages like Swedish. We release our model for further exploration and research here: https://github.com/Kungbib/swedish-bert-models .

KLEJ: Comprehensive Benchmark for Polish Language Understanding

Abstract: In recent years, a series of Transformer-based models unlocked major improvements in general natural language understanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which allow for a fair comparison of the proposed methods. However, such benchmarks are available only for a handful of languages. To alleviate this issue, we introduce a comprehensive multi-task benchmark for the Polish language understanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing datasets for named entity recognition, question-answering, textual entailment, and others. We also introduce a new sentiment analysis task for the e-commerce domain, named Allegro Reviews (AR). To ensure a common evaluation scheme and promote models that generalize to different NLU tasks, the benchmark includes datasets from varying domains and applications. Additionally, we release HerBERT, a Transformer-based model trained specifically for the Polish language, which has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an extensive evaluation, including several standard baselines and recently proposed, multilingual Transformer-based models.

RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model

Abstract: We present RobeCzech, a monolingual RoBERTa language representation model trained on Czech data. RoBERTa is a robustly optimized Transformer-based pretraining approach. We show that RobeCzech considerably outperforms equally-sized multilingual and Czech-trained contextualized language representation models, surpasses current state of the art in all five evaluated NLP tasks and reaches state-of-the-art results in four of them. The RobeCzech model is released publicly at https://hdl.handle.net/11234/1-3691 and https://huggingface.co/ufal/robeczech-base.

Summary of monolingual T-PTLMs.

A summary of the various monolingual model is presented in Tables {{table:ef8bcd93-9167-4fc2-98ae-18a752684b50}}  and {{table:80908966-b37a-4382-b2bb-ad7d717ea6e5}} . Monolingual models are pretrained based on standard model architectures like GPT {{cite:edf566d}} BERT {{cite:b3c7adc}}, {{cite:0a33ffa}}, {{cite:5333341}}, {{cite:268a20b}}, {{cite:27ae3a9}}, {{cite:e38125f}}, {{cite:f67c937}}, {{cite:cead8ff}}, {{cite:6102fa5}}, {{cite:e7bdd06}}, {{cite:7621373}}, {{cite:070d88f}}, {{cite:fb6531b}}, {{cite:cae8df3}}, {{cite:92d7b21}}, {{cite:8fa085d}}, RoBERTa {{cite:cead8ff}}, {{cite:107448c}}, {{cite:b77424f}}, {{cite:ced223a}}, {{cite:0b94237}}, {{cite:b9bfbb6}}, {{cite:a37d01c}}, ALBERT {{cite:5333341}}, {{cite:02e36b4}}, ELECTRA {{cite:2002fd7}}, {{cite:6a22200}}, and T5 {{cite:d239a47}}. As the availability of corpus from one source is limited in the case of many languages, most of these models are pretrained using corpus gathered from multiple sources. For example, IndoBERT {{cite:268a20b}} is pretrained on corpus having text from Wikipedia, News domain, and Internet. Except for models like PTT5 {{cite:d239a47}}, RuBERT {{cite:b3c7adc}} and PortugueseBERT {{cite:0a33ffa}}, all other monolingual models are pretrained from scratch. Models like PTT5, RuBERT, and PortugueseBERT are initialized from existing models and further pretrained with new language-specific vocabulary. In these models, only the transformer encoder layer parameters are copied from existing models while embedding layer parameters are randomly initialized. During CPT, embedding layer parameters are updated along with other layers.


Query Expansion Based on Crowd Knowledge for Code Search

Abstract: As code search is a frequent developer activity in software development practices, improving the performance of code search is a critical task. In the text retrieval based search techniques employed in the code search, the term mismatch problem is a critical language issue for retrieval effectiveness. By reformulating the queries, query expansion provides effective ways to solve the term mismatch problem. In this paper, we propose Query Expansion based on Crowd Knowledge (QECK), a novel technique to improve the performance of code search algorithms. QECK identifies software-specific expansion words from the high quality pseudo relevance feedback question and answer pairs on Stack Overflow to automatically generate the expansion queries. Furthermore, we incorporate QECK in the classic Rocchio's model, and propose QECK based code search method QECKRocchio. We conduct three experiments to evaluate our QECK technique and investigate QECKRocchio in a large-scale corpus containing real-world code snippets and a question and answer pair collection. The results show that QECK improves the performance of three code search algorithms by up to 64 percent in Precision, and 35 percent in NDCG. Meanwhile, compared with the state-of-the-art query expansion method, the improvement of QECK Rocchio is 22 percent in Precision, and 16 percent in NDCG.

Effective Reformulation of Query for Code Search using Crowdsourced Knowledge and Extra-Large Data Analytics

Abstract: Software developers frequently issue generic natural language queries for code search while using code search engines (e.g., GitHub native search, Krugle). Such queries often do not lead to any relevant results due to vocabulary mismatch problems. In this paper, we propose a novel technique that automatically identifies relevant and specific API classes from Stack Overflow Q & A site for a programming task written as a natural language query, and then reformulates the query for improved code search. We first collect candidate API classes from Stack Overflow using pseudo-relevance feedback and two term weighting algorithms, and then rank the candidates using Borda count and semantic proximity between query keywords and the API classes. The semantic proximity has been determined by an analysis of 1.3 million questions and answers of Stack Overflow. Experiments using 310 code search queries report that our technique suggests relevant API classes with 48% precision and 58% recall which are 32% and 48% higher respectively than those of the state-of-the-art. Comparisons with two state-of-the-art studies and three popular search engines (e.g., Google, Stack Overflow, and GitHub native search) report that our reformulated queries (1) outperform the queries of the state-of-the-art, and (2) significantly improve the code search results provided by these contemporary search engines.

Overview of approaches for preprocessing and expansion of queries.

To address the limitations of user-provided queries, approaches for preprocessing and expanding queries have been developed.
We discuss these approaches by focusing on three dimensions:
(i) the user interface, i.e., if and how a user gets involved in modifying queries,
(ii) the information used to modify queries, i.e., what additional source of knowledge an approach consults, and
(iii) the actual technique used to modify queries.
Table {{table:0562003f-763a-4fa2-970f-b0ad32ced01b}}  summarizes different approaches along these three dimensions, and we discuss them in detail in the following.


Text Classification and Sentiment Analysis

Process and extracting meaningful information from unstructured textual data using NLP techniques.

Extract structured information from clinical narratives, such as diagnoses, medications, and procedures.

Classify clinical text into categories or analyze sentiment for a comprehensive understanding.

A Review of the Trends and Challenges in Adopting Natural Language Processing Methods for Education Feedback Analysis

Abstract: Artificial Intelligence (AI) is a fast-growing area of study that stretching its presence to many business and research domains. Machine learning, deep learning, and natural language processing (NLP) are subsets of AI to tackle different areas of data processing and modelling. This review article presents an overview of AI’s impact on education outlining with current opportunities. In the education domain, student feedback data is crucial to uncover the merits and demerits of existing services provided to students. AI can assist in identifying the areas of improvement in educational infrastructure, learning management systems, teaching practices and study environment. NLP techniques play a vital role in analyzing student feedback in textual format. This research focuses on existing NLP methodologies and applications that could be adapted to educational domain applications like sentiment annotations, entity annotations, text summarization, and topic modelling. Trends and challenges in adopting NLP in education were reviewed and explored. Context-based challenges in NLP like sarcasm, domain-specific language, ambiguity, and aspect-based sentiment analysis are explained with existing methodologies to overcome them. Research community approaches to extract the semantic meaning of emoticons and special characters in feedback which conveys user opinion and challenges in adopting NLP in education are explored.

Comprehend Medical: A Named Entity Recognition and Relationship Extraction Web Service

Abstract: Comprehend Medical is a stateless and Health Insurance Portability and Accountability Act (HIPAA) eligible Named Entity Recognition (NER) and Relationship Extraction (RE) service launched under Amazon Web Services (AWS) trained using state-of-the-art deep learning models. Contrary to many existing open source tools, Comprehend Medical is scalable and does not require steep learning curve, dependencies, pipeline configurations, or installations. Currently, Comprehend Medical performs NER in five medical categories: Anatomy, Medical Condition, Medications, Protected Health Information (PHI) and Treatment, Test and Procedure (TTP). Additionally, the service provides relationship extraction for the detected entities as well as contextual information such as negation and temporality in the form of traits. Comprehend Medical provides two Application Programming Interfaces (API): 1) the NERe API which returns all the extracted named entities, their traits and the relationships between them and 2) the PHId API which returns just the protected health information contained in the text. Furthermore, Comprehend Medical is accessible through AWS Console, Java and Python Software Development Kit (SDK), making it easier for non-developers and developers to use.

Progress Notes Classification and Keyword Extraction using Attention-based Deep Learning Models with BERT

Abstract: Various deep learning algorithms have been developed to analyze different types of clinical data including clinical text classification and extracting information from 'free text' and so on. However, automate the keyword extraction from the clinical notes is still challenging. The challenges include dealing with noisy clinical notes which contain various abbreviations, possible typos, and unstructured sentences. The objective of this research is to investigate the attention-based deep learning models to classify the de-identified clinical progress notes extracted from a real-world EHR system. The attention-based deep learning models can be used to interpret the models and understand the critical words that drive the correct or incorrect classification of the clinical progress notes. The attention-based models in this research are capable of presenting the human interpretable text classification models. The results show that the fine-tuned BERT with the attention layer can achieve a high classification accuracy of 97.6%, which is higher than the baseline fine-tuned BERT classification model. In this research, we also demonstrate that the attention-based models can identify relevant keywords that are strongly related to the clinical progress note categories.

Multimodal Fusion - NLP techniques

NLP techniques provide valuable capabilities in extracting, processing, and integrating textual information within multimodal medical data fusion. They improve the comprehension of clinical text, enable the incorporation of clinical knowledge, and facilitate comprehensive and effective analysis in smart healthcare applications. Table {{table:cf88462b-41fa-472e-bb7a-45eb4349c1aa}}  presents the NLP techniques that can be adopted for multimodal fusion. Additionally, Figure {{figure:656d61cf-3aec-408f-9476-7774674ab7ee}}  illustrates the taxonomy of approaches discussed for multimodal fusion in the context of smart healthcare.


Greedy Layerwise Learning Can Scale to ImageNet

Abstract: Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image classification tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems lead to a CNN that exceeds AlexNet performance on ImageNet. Extending this training methodology to construct individual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds several members of the VGG model family on ImageNet, and can train a VGG-11 model to the same accuracy as end-to-end learning. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We illustrate several interesting properties of these models theoretically and conduct a range of experiments to study the properties this training induces on the intermediate layers.

Decoupled Greedy Learning of CNNs

Abstract: A commonly cited inefficiency of neural network training by back-propagation is the update locking problem: each layer must wait for the signal to propagate through the full network before updating. Several alternatives that can alleviate this issue have been proposed. In this context, we consider a simpler, but more effective, substitute that uses minimal feedback, which we call Decoupled Greedy Learning (DGL). It is based on a greedy relaxation of the joint training objective, recently shown to be effective in the context of Convolutional Neural Networks (CNNs) on large-scale image classification. We consider an optimization of this objective that permits us to decouple the layer training, allowing for layers or modules in networks to be trained with a potentially linear parallelization in layers. With the use of a replay buffer we show this approach can be extended to asynchronous settings, where modules can operate with possibly large communication delays. We show theoretically and empirically that this approach converges. Then, we empirically find that it can lead to better generalization than sequential greedy optimization. We demonstrate the effectiveness of DGL against alternative approaches on the CIFAR-10 dataset and on the large-scale ImageNet dataset.

Decoupled Greedy Learning of CNNs

Decoupled Greedy Learning of CNNs

Training Neural Networks with Local Error Signals

Abstract: Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network with layer-wise loss functions. In this paper we demonstrate, for the first time, that layer-wise training can approach the state-of-the-art on a variety of image datasets. We use single-layer sub-networks and two different supervised loss functions to generate local error signals for the hidden layers, and we show that the combination of these losses help with optimization in the context of local learning. Using local errors could be a step towards more biologically plausible deep learning because the global error does not have to be transported back to hidden layers. A completely backprop free variant outperforms previously reported results among methods aiming for higher biological plausibility. Code is available this https URL

Modularizing Deep Learning via Pairwise Learning With Kernels

Abstract: By redefining the conventional notions of layers, we present an alternative view on finitely wide, fully trainable deep neural networks as stacked linear models in feature spaces, leading to a kernel machine interpretation. Based on this construction, we then propose a provably optimal modular learning framework for classification that does not require between-module backpropagation. This modular approach brings new insights into the label requirement of deep learning (DL). It leverages only implicit pairwise labels (weak supervision) when learning the hidden modules. When training the output module, on the other hand, it requires full supervision but achieves high label efficiency, needing as few as ten randomly selected labeled examples (one from each class) to achieve 94.88% accuracy on CIFAR-10 using a ResNet-18 backbone. Moreover, modular training enables fully modularized DL workflows, which then simplify the design and implementation of pipelines and improve the maintainability and reusability of models. To showcase the advantages of such a modularized workflow, we describe a simple yet reliable method for estimating reusability of pretrained modules as well as task transferability in a transfer learning setting. At practically no computation overhead, it precisely described the task space structure of 15 binary classification tasks from CIFAR-10.

A Theoretical Framework for Target Propagation

Abstract: The success of deep learning, a brain-inspired form of AI, has sparked interest in understanding how the brain could similarly learn across multiple layers of neurons. However, the majority of biologically-plausible learning algorithms have not yet reached the performance of backpropagation (BP), nor are they built on strong theoretical foundations. Here, we analyze target propagation (TP), a popular but not yet fully understood alternative to BP, from the standpoint of mathematical optimization. Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP. Furthermore, our analysis reveals a fundamental limitation of difference target propagation (DTP), a well-known variant of TP, in the realistic scenario of non-invertible neural networks. We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training, while simultaneously introducing architectural flexibility by allowing for direct feedback connections from the output to each hidden layer. Our theory is corroborated by experimental results that show significant improvements in performance and in the alignment of forward weight updates with loss gradients, compared to DTP.

Learning to solve the credit assignment problem

Abstract: Backpropagation is driving today's artificial neural networks (ANNs). However, despite extensive research, it remains unclear if the brain implements this algorithm. Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspecific feedback signals to observe their effect on the cost and thus approximate their gradient. However, the convergence rate of such learning scales poorly with the number of involved neurons. Here we propose a hybrid learning approach. Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide. We provide proof that our approach converges to the true gradient for certain classes of networks. In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match or the performance of exact gradient-based learning. Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules.

Performant instantiations of reviewed methods and best reported test accuracy on standard benchmarking datasets. Note that each result entry should only be compared against the corresponding E2EBP baseline due to the potentially different test settings across entries. VGG-11B and VGG-8B are customized VGG-11 and VGG-8, respectively. Only Proxy Objective methods can match the performance of E2EBP on competitive networks.

The goal of this paper is exactly to review existing work on provably optimal avenues to train deep neural networks without E2EBP, so as to push the field to new highs.
These methods can be categorized into three distinct abstract algorithms: Proxy Objective (modular), Target Propagation (weakly modular), and Synthetic Gradients (weakly modular) An important family of E2EBP-free training methods that we leave out here is the auxiliary variables methods {{cite:52a8ea7}}, {{cite:605cb43}}, {{cite:a3b4170}}, {{cite:3b8578a}}, {{cite:75bceca}}, {{cite:e35f472}}, {{cite:758fc39}}, {{cite:09e1f94}}, {{cite:ee35001}}, {{cite:6b0de38}}, {{cite:d300835}}. These methods introduce auxiliary trainable variables that approximate the hidden activations in order to achieve parallelized training. Despite the strong theoretical guarantees, the introduced auxiliary variables may pose scalability issues and, more importantly, these methods require special, often tailor-made alternating solvers. And none of the existing work in this area has scaled these methods to beyond toy models and toy data. Therefore, we leave out these methods in this paper and interested readers may refer to the cited papers for more details..
And for each abstract algorithm, we present some popular instantiations, optimality guarantees, advantages and limitations, and potential implications on other research areas.
Some future research directions will also be sketched out.
A summary of all methods is provided in Table {{table:2296973d-edb3-4027-93bb-a9b8afd17bef}}  and their best reported results on standard benchmark datasets are listed in {{table:dc0506c7-0c98-41da-b614-360b3f8065bd}} .
And illustrations are given in Fig. {{figure:e1206729-fb05-4a6a-9df7-7d9ebd79e1bf}} .


Tailored to meet specific energy budgets at both the UAV and GNs

Optimized for collision-free paths in emergency scenarios

Significantly improved compared to other schemes based on specific requirements

Tailored to meet specific mission requirements

Maximized minimum UAV data collection throughput

UAV Trajectory Planning for AoI-Minimal Data Collection in UAV-Aided IoT Networks by Transformer

Abstract: Maintaining freshness of data collection in Internet-of-Things (IoT) networks has attracted increasing attention. By taking into account age-of-information (AoI), we investigate the trajectory planning problem of an unmanned aerial vehicle (UAV) that is used to aid a cluster-based IoT network. An optimization problem is formulated to minimize the total AoI of the collected data by the UAV from the ground IoT network. Since the total AoI of the IoT network depends on the flight time of the UAV and the data collection time at hovering points, we jointly optimize the selection of hovering points and the visiting order to these points. We exploit the state-of-the-art transformer and the weighted A*, which is a path search algorithm, to design a machine learning algorithm to solve the formulated problem. The whole UAV-IoT system is fed into the encoder network of the proposed algorithm, and the algorithm's decoder network outputs the visiting order to ground clusters. Then, the weighted A* is used to find the hovering point for each cluster in the ground IoT network. Simulation results show that the trained model by the proposed algorithm has a good generalization ability to generate solutions for IoT networks with different numbers of ground clusters, without the need to retrain the model. Furthermore, results show that our proposed algorithm can find better UAV trajectories with the minimum total AoI when compared to other algorithms.

Energy-Constrained UAV Data Collection Systems: NOMA and OMA

Abstract: This paper investigates unmanned aerial vehicle (UAV) data collection systems with different multiple access schemes, where a rotary-wing UAV is dispatched to collect data from multiple ground nodes (GNs). Our goal is to maximize the minimum UAV data collection throughput from GNs for both orthogonal multiple access (OMA) and non-orthogonal multiple access (NOMA) transmission, subject to the energy budgets at both the UAV and GNs, namely \emph{double energy limitations}. 1) For OMA, we propose an efficient algorithm by invoking alternating optimization (AO) method, where each subproblem is alternately solved by applying successive convex approximation (SCA) technique. 2) For NOMA, we first handle subproblems with fixed decoding order using SCA technique. Then, we develop a penalty-based algorithm to solve the decoding order design subproblem. Numerical results show that: i) The proposed algorithms are capable of improving the max-min throughput performance compared with other benchmark schemes; and ii) NOMA yields a higher performance gain than OMA when GNs have sufficient energy.

Hybrid Path Planning for Efficient Data Collection in UAV-Aided WSNs for Emergency Applications

Abstract: In unmanned aerial vehicle (UAV)-aided wireless sensor networks (UWSNs), a UAV is employed as a mobile sink to gather data from sensor nodes. Incorporating UAV helps prolong the network lifetime and avoid the energy-hole problem faced by sensor networks. In emergency applications, timely data collection from sensor nodes and transferal of the data to the base station (BS) is a prime requisite. The timely and safe path of UAV is one of the fundamental premises for effective UWSN operations. It is essential and challenging to identify a suitable path in an environment comprising various obstacles and to ensure that the path can efficiently reach the target point. This paper proposes a hybrid path planning (HPP) algorithm for efficient data collection by assuring the shortest collision-free path for UAV in emergency environments. In the proposed HPP scheme, the probabilistic roadmap (PRM) algorithm is used to design the shortest trajectory map and the optimized artificial bee colony (ABC) algorithm to improve different path constraints in a three-dimensional environment. Our simulation results show that the proposed HPP outperforms the PRM and conventional ABC schemes significantly in terms of flight time, energy consumption, convergence time, and flight path.

Comparison of different data collection schemes

Advantages and Disadvantages of Each Method

In the realm of UAV systems, data collection is a critical aspect that can be approached in various ways, each with its own advantages and potential drawbacks.
Table  {{table:1ac1d7e4-b98e-4f9b-a689-ade053915c85}}  compares various approaches under different data collecting modes based on different performance metrics. Static data collection schemes, such as {{cite:0ab7401}} and {{cite:43a9537}}, are ideal for scenarios where minimizing the AoI is crucial or where maximizing the minimum UAV data collection throughput from Ground Nodes (GNs) is important, subject to the energy budgets at both the UAV and GNs. On the other hand, mobile data collection schemes like {{cite:12d4cf3}} and {{cite:69aad43}} are beneficial in scenarios where the IoT devices are widely dispersed or when the area to be covered is large. These schemes are particularly effective in smart city environments where secure data collection is a priority. Hybrid or collaborative data collection schemes combine the characteristics of both static and mobile modes. Schemes such as {{cite:3e05d51}} and {{cite:1eeed1a}} are more energy-efficient than the static mode but less so than the mobile mode. These schemes are particularly effective in emergency or underwater environments where efficient data gathering is crucial. Therefore, choosing an appropriate scheme requires careful consideration of various factors such as AoI, energy efficiency, flight time, throughput, and specific application scenarios.


Universal Sentence Encoder

Abstract: We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate performance on diverse transfer tasks. Two variants of the encoding models allow for trade-offs between accuracy and compute resources. For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance. Comparisons are made with baselines that use word level transfer learning via pretrained word embeddings as well as baselines do not use any transfer learning. We find that transfer learning using sentence embeddings tends to outperform word level transfer. With transfer learning via sentence embeddings, we observe surprisingly good performance with minimal amounts of supervised training data for a transfer task. We obtain encouraging results on Word Embedding Association Tests (WEAT) targeted at detecting model bias. Our pre-trained sentence encoding models are made freely available for download and on TF Hub.

An efficient framework for learning sentence representations

Abstract: In this work we propose a simple and efficient framework for learning sentence representations from unlabelled data. Drawing inspiration from the distributional hypothesis and recent work on learning sentence representations, we reformulate the problem of predicting the context in which a sentence appears as a classification problem. Given a sentence and its context, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations. This allows us to efficiently learn different types of encoding functions, and we show that the model learns high-quality sentence representations. We demonstrate that our sentence representations outperform state-of-the-art unsupervised and supervised representation learning methods on several downstream NLP tasks that involve understanding sentence semantics while achieving an order of magnitude speedup in training time.

Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features

Abstract: The recent tremendous success of unsupervised word embeddings in a multitude of applications raises the obvious question if similar methods could be derived to improve embeddings (i.e. semantic representations) of word sequences as well. We present a simple but efficient unsupervised objective to train distributed representations of sentences. Our method outperforms the state-of-the-art unsupervised models on most benchmark tasks, highlighting the robustness of the produced general-purpose sentence embeddings.

Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings

Abstract: We consider the problem of learning general-purpose, paraphrastic sentence embeddings, revisiting the setting of Wieting et al. (2016b). While they found LSTM recurrent networks to underperform word averaging, we present several developments that together produce the opposite conclusion. These include training on sentence pairs rather than phrase pairs, averaging states to represent sequences, and regularizing aggressively. These improve LSTMs in both transfer learning and supervised settings. We also introduce a new recurrent architecture, the Gated Recurrent Averaging Network, that is inspired by averaging and LSTMs while outperforming them both. We analyze our learned models, finding evidence of preferences for particular parts of speech and dependency relations.

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Abstract: Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features. Efforts to obtain embeddings for larger chunks of text, such as sentences, have however not been so successful. Several attempts at learning unsupervised representations of sentences have not reached satisfactory enough performance to be widely adopted. In this paper, we show how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks. Much like how computer vision uses ImageNet to obtain features, which can then be transferred to other tasks, our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks. Our encoder is publicly available.

ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations

Abstract: We describe PARANMT-50M, a dataset of more than 50 million English-English sentential paraphrase pairs. We generated the pairs automatically by using neural machine translation to translate the non-English side of a large parallel corpus, following Wieting et al. (2017). Our hope is that ParaNMT-50M can be a valuable resource for paraphrase generation and can provide a rich source of semantic knowledge to improve downstream natural language understanding tasks. To show its utility, we use ParaNMT-50M to train paraphrastic sentence embeddings that outperform all supervised systems on every SemEval semantic textual similarity competition, in addition to showing how it can be used for paraphrase generation.

Towards Universal Paraphrastic Sentence Embeddings

Abstract: We consider the problem of learning general-purpose, paraphrastic sentence embeddings based on supervision from the Paraphrase Database (Ganitkevitch et al., 2013). We compare six compositional architectures, evaluating them on annotated textual similarity datasets drawn both from the same distribution as the training data and from a wide range of other domains. We find that the most complex architectures, such as long short-term memory (LSTM) recurrent neural networks, perform best on the in-domain data. However, in out-of-domain scenarios, simple architectures such as word averaging vastly outperform LSTMs. Our simplest averaging model is even competitive with systems tuned for the particular tasks while also being extremely efficient and easy to use. In order to better understand how these architectures compare, we conduct further experiments on three supervised NLP tasks: sentence similarity, entailment, and sentiment classification. We again find that the word averaging models perform well for sentence similarity and entailment, outperforming LSTMs. However, on sentiment classification, we find that the LSTM performs very strongly-even recording new state-of-the-art performance on the Stanford Sentiment Treebank. We then demonstrate how to combine our pretrained sentence embeddings with these supervised tasks, using them both as a prior and as a black box feature extractor. This leads to performance rivaling the state of the art on the SICK similarity and entailment tasks. We release all of our resources to the research community with the hope that they can serve as the new baseline for further work on universal sentence embeddings.

Skip-Thought Vectors

Abstract: We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice. We will make our encoder publicly available.

Transfer learning settings used by sentence encoders on semantic similarity tasks in recent years, ranked by time of publication. UE: unsupervised evaluation; FT: feature transfer; NT: network transfer.

Although most sentence encoders have been built with the goal of transferring to downstream tasks, the methodology of transfer learning itself is inconsistent {{cite:6e7cb0c}}. There has not been a consensus on which transfer learning setting to use with sentence encoders for semantic similarity tasks (Table {{table:b109e675-3c13-4f5f-8207-d43ed9543cc1}} ). The two semantic similarity datasets popularly used for sentence embedding evaluation are SICK Relatedness and SemEval STS. Among sentence encoders that evaluate on these two tasks within three years, SIF on GloVE {{cite:6cb3790}} and ParaNMT {{cite:a8e0394}} use unsupervised evaluation on STS but use feature transfer on SICK; InferSent {{cite:5dc1090}} use feature transfer on both; GRAN {{cite:be9338d}} and Sent2vec {{cite:1d2502e}} use unsupervised evaluation on both; Paragram {{cite:40f85f2}} is the only work, to our knowlege, that experiments on both feature transfer and network transfer for both datasets; Cer et al. cer2018universal use unsupervised evaluation while Logeswaran and Lee logeswaran2018efficient use feature transfer on SICK, but they do not evaluate on STS.


[leftmargin=*]
2350-Dimensional feature vector

Hashing trick to collapse into a vector of fixed size

[leftmargin=*]
Import function feature

General information feature

Byte entropy features

2478-D feature vector

[leftmargin=*]
ACER with DQN learns both a policy model and a Q-function

Boltzman exploration and exploitation where mutation are proportional to expected Q-value

Mutations till evasion or 10 rounds

[leftmargin=*]
Novel static feature extraction

RLAttackNet using DQN and optimized using double and dueling DQN

Different Q-network for choosing best action and Q-value

Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning

Abstract: Machine learning is a popular approach to signatureless malware detection because it can generalize to never-before-seen malware families and polymorphic strains. This has resulted in its practical use for either primary detection engines or for supplementary heuristic detection by anti-malware vendors. Recent work in adversarial machine learning has shown that deep learning models are susceptible to gradient-based attacks, whereas non-differentiable models that report a score can be attacked by genetic algorithms that aim to systematically reduce the score. We propose a more general framework based on reinforcement learning (RL) for attacking static portable executable (PE) anti-malware engines. The general framework does not require a differentiable model nor does it require the engine to produce a score. Instead, an RL agent is equipped with a set of functionality-preserving operations that it may perform on the PE file. Through a series of games played against the anti-malware engine, it learns which sequences of operations are likely to result in evading the detector for any given malware sample. This enables completely black-box attacks against static PE anti-malware, and produces functional evasive malware samples as a direct result. We show in experiments that our method can attack a gradient-boosted machine learning model with evasion rates that are substantial and appear to be strongly dependent on the dataset. We demonstrate that attacks against this model appear to also evade components of publicly hosted antivirus engines. Adversarial training results are also presented: by retraining the model on evasive ransomware samples, a subsequent attack is 33% less effective. However, there are overfitting dangers when adversarial training, which we note. We release code to allow researchers to reproduce and improve this approach.

DeepDetectNet vs RLAttackNet: An adversarial method to improve deep learning-based static malware detection model

Abstract: Deep learning methods are being increasingly widely used in static malware detection field because they can summarize the feature of malware and its variants that have never appeared before. But similar to the picture recognition model, the static malware detection model based on deep learning is also vulnerable to the interference of adversarial samples. When the input feature vectors of the malware detection model is based on static features of Windows PE (Portable Executable, PE) file, the model is vulnerable to gradient-based attacks. Regarding the issue above, a method of adversarial sample generation is proposed, which can summarize the blind spots of the original detection model. However, the existing malware adversarial sample generation method is not universal and low in generation efficiency due to the need for human control and difficulty in maintaining a normal file format. In response to these problems, this paper proposes a novel method of automatic adversarial samples generation based on deep reinforcement learning. Firstly, a static PE malware detection model based on deep learning called DeepDetectNet is constructed, the original AUC of which can reach 0.989. Then, an adversarial sample generation model based on reinforcement learning called RLAttackNet is implemented, which generates malware samples that can bypass DeepDetectNet. Finally, when we re-input the adversarial samples into the previously trained DeepDetectNet, the original defects of DeepDetectNet can be reinforced. Experimental results show that the RLAttackNet proposed in this paper can generate about 19.13% of malware samples bypass DeepDetectNet. When DeepDetectNet is retrained with these adversarial samples, the AUC value improves from 0.989 to 0.996 and attack success rate has a significant drop, from 19.13% to 3.1%, compared with the original model.

A reinforcement learning adversarial attacks.

To counter the need of differentiable model for gradient based approaches, reinforcement learning agent has been proposed to generate adversarial against malware detection. RL agent is provided with a set of operations to modify PE files while also preserving the functionality of malware. Goal of RL agent will be to perform sequence of operations on malware to evade detection. Reinforcement learning enables complete blackbox attacks to detector, creating real world attack scenario where attacker is completely unknown about detector. This reinforcement learning process is built around Markov decision process as shown in Figure {{figure:6024531d-2400-4080-bdef-4d78f661c6c1}} . Table {{table:1605bc3e-e66a-48f2-b804-2d0144c804d1}}  provides comparison of all RL approaches on adversarial evasion attack for Windows malware. Anderson et al.  {{cite:cf19f2b}} proposed a whitepaper on evading malware detection by modifying Windows PE bytes for the first time. Anderson et al. {{cite:f6a6287}} extended results of work done in {{cite:cf19f2b}} to perform generic black box attacks on static PE malware detection without assuming any knowledge of detector model's structure and features, retrieving only malicious/benign label. Actor Critic Model with Experience Replay (ACER) is used to learn both policy model {{formula:2b792ded-017f-476e-a6c9-480e3abc97ed}}  and a Q-function to estimate the state-action value. Reward of 0 is provided for malware samples which are detected by anti-malware engine and 10 for ones that can evade detection. Reward and state after each action are provided to an agent as a feedback. Feature vector is prepared such that it summarizes the state of the environment. 2350- dimensional feature vector is extracted from Windows PE malware consisting of features as:


AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles

Abstract: Developing and testing algorithms for autonomous vehicles in real world is an expensive and time consuming process. Also, in order to utilize recent advances in machine intelligence and deep learning we need to collect a large amount of annotated training data in a variety of conditions and environments. We present a new simulator built on Unreal Engine that offers physically and visually realistic simulations for both of these goals. Our simulator includes a physics engine that can operate at a high frequency for real-time hardware-in-the-loop (HITL) simulations with support for popular protocols (e.g. MavLink). The simulator is designed from the ground up to be extensible to accommodate new types of vehicles, hardware platforms and software protocols. In addition, the modular design enables various components to be easily usable independently in other projects. We demonstrate the simulator by first implementing a quadrotor as an autonomous vehicle and then experimentally comparing the software components with real-world flights.

Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

Abstract: We present a photo-realistic training and evaluation simulator (Sim4CV) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous driving using supervised learning. The simulator fully integrates both several state-of-the-art tracking algorithms with a benchmark evaluation tool and a deep neural network (DNN) architecture for training vehicles to drive autonomously. It generates synthetic photo-realistic datasets with automatic ground truth annotations to easily extend existing real-world datasets and provides extensive synthetic data variety through its ability to reconfigure synthetic worlds on the fly using an automatic world generation tool. The supplementary video can be viewed a https://youtu.be/SqAxzsQ7qUU

UAVs simulation tools.

UAV simulation tools emulate virtual environments to model UAV flights in close-to-reality situations. Some simulators incorporate the Motion of Capture (MOCAP), which allows simulating UAVs' natural movements {{cite:c8206ab}}.
Another tool is MAVLink, a lightweight messaging protocol for communicating with drones to test communication protocols and algorithms. Software In The Loop (SITL) is a hardware-free simulation environment that facilitates simulating real-time UAV operations. It includes a c++ code to directly implement autopilot operation on the user's computer for testing {{cite:9f3e32e}}. The list of tools for simulating UAV networking is large and still growing. A comparative analysis of some popular simulation tools, including XPlane {{cite:5b59d69}}, FlightGear {{cite:d7d1b03}} (compatible with MATLAB Simulink), Gazebo {{cite:b76db82}}, JMavSim {{cite:197f085}}, Microsoft AirSim {{cite:bbb362f}}, and UE4Sim {{cite:4431b83}}, is presented in Table {{table:aededa7d-553c-46f4-a95c-c973ba00b133}} .


Texture based alignment query and support images

Texture based Prototypical Network for Few-Shot Semantic Segmentation of Forest Cover: Generalizing for Different Geographical Regions

Abstract: Forest plays a vital role in reducing greenhouse gas emissions and mitigating climate change besides maintaining the world's biodiversity. The existing satellite-based forest monitoring system utilizes supervised learning approaches that are limited to a particular region and depend on manually annotated data to identify forest. This work envisages forest identification as a few-shot semantic segmentation task to achieve generalization across different geographical regions. The proposed few-shot segmentation approach incorporates a texture attention module in the prototypical network to highlight the texture features of the forest. Indeed, the forest exhibits a characteristic texture different from other classes, such as road, water, etc. In this work, the proposed approach is trained for identifying tropical forests of South Asia and adapted to determine the temperate forest of Central Europe with the help of a few (one image for 1-shot) manually annotated support images of the temperate forest. An IoU of 0.62 for forest class (1-way 1-shot) was obtained using the proposed method, which is significantly higher (0.46 for PANet) than the existing few-shot semantic segmentation approach. This result demonstrates that the proposed approach can generalize across geographical regions for forest identification, creating an opportunity to develop a global forest cover identification tool.

Few-Shot Rotation-Invariant Aerial Image Semantic Segmentation

Abstract: Few-shot aerial image semantic segmentation is a challenging task that requires precisely parsing unseen-category objects in query aerial images with limited annotated support aerial images. Formally, category prototypes would be extracted from support samples to segment query images in a pixel-to-pixel matching manner. However, aerial objects in aerial images are often distributed with arbitrary orientations, and varying orientations could cause a dramatic feature change. This unique property of aerial images renders conventional matching manner without consideration of orientations fails to activate same-category objects with different orientations. Furthermore, the oscillation of the confidence scores in existing rotation-insensitive algorithms, engendered by the striking changes of object orientations, often leads to false recognition of lower scored rotated semantic objects. To tackle these challenges, inspired by the intrinsic rotation invariance in aerial images, we propose a novel few-shot rotation-invariant aerial semantic segmentation network (FRINet) to efficiently segment aerial semantic objects with diverse orientations. Specifically, through extracting orientation-varying yet category-consistent support information, FRINet provides rotation-adaptive matching for each query feature in a feature-aggregation manner. Meanwhile, to encourage consistent predictions for aerial objects with arbitrary orientations, segmentation predictions from different orientations are supervised by the same label and further fused to obtain the final rotation-invariant prediction in a complementary manner. Moreover, aiming at providing a better solution searching space, the backbones are newly pretrained in the base category to basically boost the segmentation performance. Extensive experiments on the few-shot aerial image semantic segmentation benchmark demonstrate that the proposed FRINet achieves a new state-of-the-art performance. The code is available at https://github.com/caoql98/FRINet.

Overview of studies addressing challenges in few-shot image segmentation for remote sensing aerial images

Few-shot image segmentation in remote sensing

Another common approach in few-shot image segmentation is to use deep learning networks, specifically convolutional neural networks (CNNs) {{cite:afa9535}}. These networks have shown great success in image segmentation tasks and have been adapted for few-shot learning scenarios. Researchers have explored different architectures and training strategies to improve the performance of CNNs in few-shot image segmentation{{cite:afa9535}}. Meta-learning, which involves training a model to learn how to learn, has also been applied to few-shot image segmentation with promising results {{cite:afa9535}}. Meta-learning algorithms aim to extract meta-knowledge from a set of tasks and use this knowledge to quickly adapt to new tasks with only a few labeled examples. In terms of applications, few-shot image segmentation in aerial images has various potential applications. One application is in urban planning, where few-shot image segmentation can be used to identify and segment different types of buildings, roads, and other urban infrastructure {{cite:5083e94}}, {{cite:3e4c8f9}}, {{cite:fbed875}}. Another application is in land-use and land-cover determination, where few-shot image segmentation can be used to classify different types of land cover, such as forests, agricultural land, and water bodies. Few-shot image segmentation can also be used in environmental monitoring and climate modeling to analyze changes in vegetation cover, water resources, and other environmental factors. In the field of wildfire recognition, detection, and segmentation, deep learning models have shown great potential{{cite:1695194}}. These models have been successfully applied to aerial and ground images to accurately classify wildfires, detect their presence, and segment the fire regions. Various deep learning architectures have been explored, including CNNs, one-stage detectors (such as YOLO), two-stage detectors (such as Faster R-CNN), and encoder-decoder models (such as U-Net and DeepLab). In the context of UAV images, a framework has been proposed for removing spatiotemporal objects from UAV images before generating the orthomosaic. The framework consists of two main processes: image segmentation and image inpainting. Image segmentation is performed using the Mask R-CNN algorithm, which detects and segments vehicles in the UAV images. The segmented areas are then masked to be removed. Image inpainting is carried out using the large mask inpainting (LaMa) method, a deep learning-based technique that reconstructs damaged or missing parts of an image {{cite:403362d}}. Additionally, a more extensive examination of aerial image-based FSIS can be found in the Table {{table:6680a60f-7483-42b4-bbf8-4be82ef588a9}} .


- Lack of decentralization feature in modern authentication schemes

- Secure authentication and privacy protection problems

- The energy and number of transactions burdens while updating the distributed ledgers and blockchain-transfer operation

- Security issues in DPoS consensus mechanism
- Potential collusion among miner candidates and attacked high-stake vehicles while choosing miner candidates by stake-based voting

- Elliptic curve cryptography (ECC) for mutual authentication

- Message authentication code
- Public-private key pair

- Controlling the number of transactions optimally by an approach called distributed clustering

- Reputation-based miner selection from candidates
- Two step verification and auditing by active and standby miners
- Contract theory

- An authentication and key-exchange scheme having lightweight, scalability, decentralization, and anonymity features

- The proposed authentication scheme ensures that it is secure from a number of common attacks
- The scheme has two properties such as decentralization and anonymity

- The proposed model consumes considerably less amount of energy and requires less number of transactions compared with the Bitcoin

- The proposed security-enhanced mechanism establishes it excellencies in terms of defending internal collusion, high detection rate of compromised candidate vehicles with better reputation scheme relative to existing schemes, and optimized the utilities of all participant miners

Blockchain-based Lightweight Authentication Mechanism for Vehicular Fog Infrastructure

Abstract: With the increasing development of advanced communication technologies, vehicles are becoming smarter and more connected. Due to the tremendous growth of various vehicular applications, a huge amount of data is generated through advanced on-board devices and is deemed critical to improve driving safety and enhance vehicular services. However, cloud based models often fall short in applications where latency and mobility are critical. In order to fully realize the potential of vehicular networks, the challenges of efficient communication and computation need to be addressed. In this direction, vehicular fog computing (VFC) has emerged which extends the concept of fog computing to conventional vehicular networks. It is a geographically distributed paradigm that has the potential to conduct time-critical and data-intensive tasks by pushing intelligence (i.e. computing resources, storage, and application services) in the vicinity of end vehicles. However secure and reliable transmission are of significant importance in highly-mobile vehicular networks in order to ensure the optimal Quality of Service (QoS). In this direction, several authentication mechanisms have been proposed in the literature but most of them are found unfit due to absence of decentralization, anonymity, and trust characteristics. Thus, an effective cross-datacenter authentication and key-exchange scheme based on blockchain and elliptic curve cryptography (ECC) is proposed in this paper. Here, the distributed ledger of blockchain is used for maintaining the network information while the highly secure ECC is employed for mutual authentication between vehicles and road side units (RSUs). Additionally, the proposed scheme is lightweight and scalable for the considered VFC setup. The performance evaluation results against the existing state-of-the-art reveal that the proposed scheme accomplishes enhanced security features.

Distributed Blockchain-Based Message Authentication Scheme for Connected Vehicles

Abstract: Vehicular ad-hoc networks (VANETs) have several security issues such as privacy preservation, secure authentication, and system reliability. In the VANET, a vehicle communicates with other vehicles or infrastructures using broadcasting messages. These messages contain not only normal traffic information, but also identification information of sender. In general, the identification information remains encrypted to ensure privacy. However, the conventional centralized system can decrypt the identification information using private information of the sender vehicle. As a result, the central server can often be targeted by adversaries. We propose a message authentication scheme for anonymity and decentralization of information using blockchain technology. Here, we introduce public-private key and message authentication code (MAC) for secure authentication. In this paper, we adopt consensus algorithms for composing blockchain system such as the proof of work (PoW) and Practical Byzantine Fault Tolerance (PBFT) into the proposed authentication process. Finally, we demonstrate that the proposed method is secure from the attacks which include impersonation from internal attacker as well as typical attacks.

An Energy-Efficient Transaction Model for the Blockchain-enabled Internet of Vehicles (IoV)

Abstract: The blockchain is a safe, reliable and innovative mechanism for managing numerous vehicles seeking connectivity. However, following the principles of the blockchain, the number of transactions required to update ledgers pose serious issues for vehicles as these may consume the maximum available energy. To resolve this, an efficient model is presented in this letter which is capable of handling the energy demands of the blockchain-enabled Internet of Vehicles (IoV) by optimally controlling the number of transactions through distributed clustering. Numerical results suggest that the proposed approach is 40.16% better in terms of energy conservation and 82.06% better in terms of the number of transactions required to share the entire blockchain-data compared with the traditional blockchain.

Towards Secure Blockchain-enabled Internet of Vehicles: Optimizing Consensus Management Using Reputation and Contract Theory

Abstract: In Internet of Vehicles (IoV), data sharing among vehicles is essential to improve driving safety and enhance vehicular services. To ensure data sharing security and traceability, highefficiency Delegated Proof-of-Stake consensus scheme as a hard security solution is utilized to establish blockchain-enabled IoV (BIoV). However, as miners are selected from miner candidates by stake-based voting, it is difficult to defend against voting collusion between the candidates and compromised high-stake vehicles, which introduces serious security challenges to the BIoV. To address such challenges, we propose a soft security enhancement solution including two stages: (i) miner selection and (ii) block verification. In the first stage, a reputation-based voting scheme for the blockchain is proposed to ensure secure miner selection. This scheme evaluates candidates' reputation by using both historical interactions and recommended opinions from other vehicles. The candidates with high reputation are selected to be active miners and standby miners. In the second stage, to prevent internal collusion among the active miners, a newly generated block is further verified and audited by the standby miners. To incentivize the standby miners to participate in block verification, we formulate interactions between the active miners and the standby miners by using contract theory, which takes block verification security and delay into consideration. Numerical results based on a real-world dataset indicate that our schemes are secure and efficient for data sharing in BIoV.

Summary of literatures on the recent works in blockchain and IoV integration challenges

In this section, we have presented the challenges associated with integration of blockchain with IoV application scenarios such as security & privacy, performance, IoV-specific & optimized consensus, and incentive mechanisms. Moreover, we have also highlighted the potential solutions to address these four challenges presented in different literatures. These works are summarized in Table {{table:ae249a93-33be-4899-ae6c-15edb568000c}} .


Attend to Medical Ontologies: Content Selection for Clinical Abstractive Summarization

Abstract: Sequence-to-sequence (seq2seq) network is a well-established model for text summarization task. It can learn to produce readable content; however, it falls short in effectively identifying key regions of the source. In this paper, we approach the content selection problem for clinical abstractive summarization by augmenting salient ontological terms into the summarizer. Our experiments on two publicly available clinical data sets (107,372 reports of MIMIC-CXR, and 3,366 reports of OpenI) show that our model statistically significantly boosts state-of-the-art results in terms of ROUGE metrics (with improvements: 2.9% RG-1, 2.5% RG-2, 1.9% RG-L), in the healthcare domain where any range of improvement impacts patients’ welfare.

CAiRE-COVID: A Question Answering and Query-focused Multi-Document Summarization System for COVID-19 Scholarly Information Management

Abstract: The outbreak of COVID-19 raises attention from the researchers from various communities. While many scientific articles have been published, a system that can provide reliable information to COVID-19 related questions from the latest academic resources is crucial, especially for the medical community in the current time-critical race to treat patients and to find a cure for the virus. To address the requests, we propose our CAiRE-COVID, a neural-based system that uses open-domain question answering (QA) techniques combined with summarization techniques for mining the available scientific literature. It leverages the Information Retrieval (IR) system and QA models to extract relevant snippets from existing literature given a query. Fluent summaries are also provided to help understand the content in a more efficient way. Our system has been awarded as winner for one of the tasks in CORD-19 Kaggle Challenge. We also launched our CAiRE-COVID website for broader use. The code for our system is also open-sourced to bootstrap further study.

Continual BERT: Continual Learning for Adaptive Extractive Summarization of COVID-19 Literature

Abstract: The scientific community continues to publish an overwhelming amount of new research related to COVID-19 on a daily basis, leading to much literature without little to no attention. To aid the community in understanding the rapidly flowing array of COVID-19 literature, we propose a novel BERT architecture that provides a brief yet original summarization of lengthy papers. The model continually learns on new data in online fashion while minimizing catastrophic forgetting, thus fitting to the need of the community. Benchmark and manual examination of its performance show that the model provide a sound summary of new scientific literature.

Graph Enhanced Contrastive Learning for Radiology Findings Summarization

Abstract: The impression section of a radiology report summarizes the most prominent observation from the findings section and is the most important section for radiologists to communicate to physicians. Summarizing findings is time-consuming and can be prone to error for inexperienced radiologists, and thus automatic impression generation has attracted substantial attention. With the encoder-decoder framework, most previous studies explore incorporating extra knowledge (e.g., static pre-defined clinical ontologies or extra background information). Yet, they encode such knowledge by a separate encoder to treat it as an extra input to their models, which is limited in leveraging their relations with the original findings. To address the limitation, we propose a unified framework for exploiting both extra knowledge and the original findings in an integrated way so that the critical information (i.e., key words and their relations) can be extracted in an appropriate way to facilitate impression generation. In detail, for each input findings, it is encoded by a text encoder, and a graph is constructed through its entities and dependency tree. Then, a graph encoder (e.g., graph neural networks (GNNs)) is adopted to model relation information in the constructed graph. Finally, to emphasize the key words in the findings, contrastive learning is introduced to map positive samples (constructed by masking non-key words) closer and push apart negative ones (constructed by masking key words). The experimental results on OpenI and MIMIC-CXR confirm the effectiveness of our proposed method.

MSˆ2: Multi-Document Summarization of Medical Studies

Abstract: To assess the effectiveness of any medical intervention, researchers must conduct a time-intensive and manual literature review. NLP systems can help to automate or assist in parts of this expensive process. In support of this goal, we release MSˆ2 (Multi-Document Summarization of Medical Studies), a dataset of over 470k documents and 20K summaries derived from the scientific literature. This dataset facilitates the development of systems that can assess and aggregate contradictory evidence across multiple studies, and is the first large-scale, publicly available multi-document summarization dataset in the biomedical domain. We experiment with a summarization system based on BART, with promising early results, though significant work remains to achieve higher summarization quality. We formulate our summarization inputs and targets in both free text and structured forms and modify a recently proposed metric to assess the quality of our system’s generated summaries. Data and models are available at https://github.com/allenai/ms2.

Readability Controllable Biomedical Document Summarization

Abstract: Different from general documents, it is recognised that the ease with which people can understand a biomedical text is eminently varied, owing to the highly technical nature of biomedical documents and the variance of readers' domain knowledge. However, existing biomedical document summarization systems have paid little attention to readability control, leaving users with summaries that are incompatible with their levels of expertise. In recognition of this urgent demand, we introduce a new task of readability controllable summarization for biomedical documents, which aims to recognise users' readability demands and generate summaries that better suit their needs: technical summaries for experts and plain language summaries (PLS) for laymen. To establish this task, we construct a corpus consisting of biomedical papers with technical summaries and PLSs written by the authors, and benchmark multiple advanced controllable abstractive and extractive summarization models based on pre-trained language models (PLMs) with prevalent controlling and generation techniques. Moreover, we propose a novel masked language model (MLM) based metric and its variant to effectively evaluate the readability discrepancy between lay and technical summaries. Experimental results from automated and human evaluations show that though current control techniques allow for a certain degree of readability adjustment during generation, the performance of existing controllable summarization methods is far from desirable in this task.

FactReranker: Fact-guided Reranker for Faithful Radiology Report Summarization

Abstract: Automatic radiology report summarization is a crucial clinical task, whose key challenge is to maintain factual accuracy between produced summaries and ground truth radiology findings. Existing research adopts reinforcement learning to directly optimize factual consistency metrics such as CheXBert or RadGraph score. However, their decoding method using greedy search or beam search considers no factual consistency when picking the optimal candidate, leading to limited factual consistency improvement. To address it, we propose a novel second-stage summarizing approach FactReranker, the first attempt that learns to choose the best summary from all candidates based on their estimated factual consistency score. We propose to extract medical facts of the input medical report, its gold summary, and candidate summaries based on the RadGraph schema and design the fact-guided reranker to efficiently incorporate the extracted medical facts for selecting the optimal summary. We decompose the fact-guided reranker into the factual knowledge graph generation and the factual scorer, which allows the reranker to model the mapping between the medical facts of the input text and its gold summary, thus can select the optimal summary even the gold summary can't be observed during inference. We also present a fact-based ranking metric (RadMRR) for measuring the ability of the reranker on selecting factual consistent candidates. Experimental results on two benchmark datasets demonstrate the superiority of our method in generating summaries with higher factual consistency scores when compared with existing methods.

Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewards

Abstract: The growth of online consumer health questions has led to the necessity for reliable and accurate question answering systems. A recent study showed that manual summarization of consumer health questions brings significant improvement in retrieving relevant answers. However, the automatic summarization of long questions is a challenging task due to the lack of training data and the complexity of the related subtasks, such as the question focus and type recognition. In this paper, we introduce a reinforcement learning-based framework for abstractive question summarization. We propose two novel rewards obtained from the downstream tasks of (i) question-type identification and (ii) question-focus recognition to regularize the question generation model. These rewards ensure the generation of semantically valid questions and encourage the inclusion of key medical entities/foci in the question summary. We evaluated our proposed method on two benchmark datasets and achieved higher performance over state-of-the-art models. The manual evaluation of the summaries reveals that the generated questions are more diverse and have fewer factual inconsistencies than the baseline summaries

Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations

Abstract: Fine-tuning pretrained models for automatically summarizing doctor-patient conversation transcripts presents many challenges: limited training data, significant domain shift, long and noisy transcripts, and high target summary variability. In this paper, we explore the feasibility of using pretrained transformer models for automatically summarizing doctor-patient conversations directly from transcripts. We show that fluent and adequate summaries can be generated with limited training data by fine-tuning BART on a specially constructed dataset. The resulting models greatly surpass the performance of an average human annotator and the quality of previous published work for the task. We evaluate multiple methods for handling long conversations, comparing them to the obvious baseline of truncating the conversation to fit the pretrained model length limit. We introduce a multistage approach that tackles the task by learning two fine-tuned models: one for summarizing conversation chunks into partial summaries, followed by one for rewriting the collection of partial summaries into a complete summary. Using a carefully chosen fine-tuning dataset, this method is shown to be effective at handling longer conversations, improving the quality of generated summaries. We conduct both an automatic evaluation (through ROUGE and two concept-based metrics focusing on medical findings) and a human evaluation (through qualitative examples from literature, assessing hallucination, generalization, fluency, and general quality of the generated summaries).

BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model

Abstract: Pretrained language models have served as important backbones for natural language processing. Recently, in-domain pretraining has been shown to benefit various domain-specific downstream tasks. In the biomedical domain, natural language generation (NLG) tasks are of critical importance, while understudied. Approaching natural language understanding (NLU) tasks as NLG achieves satisfying performance in the general domain through constrained language generation or language prompting. We emphasize the lack of in-domain generative language models and the unsystematic generative downstream benchmarks in the biomedical domain, hindering the development of the research community. In this work, we introduce the generative language model BioBART that adapts BART to the biomedical domain. We collate various biomedical language generation tasks including dialogue, summarization, entity linking, and named entity recognition. BioBART pretrained on PubMed abstracts has enhanced performance compared to BART and set strong baselines on several tasks. Furthermore, we conduct ablation studies on the pretraining tasks for BioBART and find that sentence permutation has negative effects on downstream tasks.

Abstractive summarization of hospitalisation histories with transformer networks

Abstract: In this paper we present a novel approach to abstractive summarization of patient hospitalisation histories. We applied an encoder-decoder framework with Longformer neural network as an encoder and BERT as a decoder. Our experiments show improved quality on some summarization tasks compared with pointer-generator networks. We also conducted a study with experienced physicians evaluating the results of our model in comparison with PGN baseline and human-generated abstracts, which showed the effectiveness of our model.

Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2

Abstract: With the COVID-19 pandemic, there is a growing urgency for medical community to keep up with the accelerating growth in the new coronavirus-related literature. As a result, the COVID-19 Open Research Dataset Challenge has released a corpus of scholarly articles and is calling for machine learning approaches to help bridging the gap between the researchers and the rapidly growing publications. Here, we take advantage of the recent advances in pre-trained NLP models, BERT and OpenAI GPT-2, to solve this challenge by performing text summarization on this dataset. We evaluate the results using ROUGE scores and visual inspection. Our model provides abstractive and comprehensive information based on keywords extracted from the original articles. Our work can help the the medical community, by providing succinct summaries of articles for which the abstract are not already available.

Automated Lay Language Summarization of Biomedical Scientific Reviews

Abstract: Health literacy has emerged as a crucial factor in making appropriate health decisions and ensuring treatment outcomes. However, medical jargon and the complex structure of professional language in this domain make health information especially hard to interpret. Thus, there is an urgent unmet need for automated methods to enhance the accessibility of the biomedical literature to the general population. This problem can be framed as a type of translation problem between the language of healthcare professionals, and that of the general public. In this paper, we introduce the novel task of automated generation of lay language summaries of biomedical scientific reviews, and construct a dataset to support the development and evaluation of automated methods through which to enhance the accessibility of the biomedical literature. We conduct analyses of the various challenges in solving this task, including not only summarization of the key points but also explanation of background knowledge and simplification of professional language. We experiment with state-of-the-art summarization models as well as several data augmentation techniques, and evaluate their performance using both automated metrics and human assessment. Results indicate that automatically generated summaries produced using contemporary neural architectures can achieve promising quality and readability as compared with reference summaries developed for the lay public by experts (best ROUGE-L of 50.24 and Flesch-Kincaid readability score of 13.30). We also discuss the limitations of the current attempt, providing insights and directions for future work.

Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization

Abstract: We consider the problem of automatically generating a narrative biomedical evidence summary from multiple trial reports. We evaluate modern neural models for abstractive summarization of relevant article abstracts from systematic reviews previously conducted by members of the Cochrane collaboration, using the authors conclusions section of the review abstract as our target. We enlist medical professionals to evaluate generated summaries, and we find that summarization systems yield consistently fluent and relevant synopses, but these often contain factual inaccuracies. We propose new approaches that capitalize on domain-specific models to inform summarization, e.g., by explicitly demarcating snippets of inputs that convey key findings, and emphasizing the reports of large and high-quality trials. We find that these strategies modestly improve the factual accuracy of generated summaries. Finally, we propose a new method for automatically evaluating the factuality of generated narrative evidence syntheses using models that infer the directionality of reported findings.

Overview of PLMs-based Methods. Besides how they use PLMs and what PLMs they used, we also list other features of these methods, according to types of output summary (extractive, abstractive, hybrid), numbers of input documents (single, multiple), and types of input documents (biomedical literature summarization, radiology report summarization, medical dialogue summarization, and medical question summarization). "-” means datasets that are not released.

the feature-based methods independently utilize contextual representations from PLMs without refining the pre-trained parameters of PLMs.
The representations from PLMs are generally concatenated with the representations from the encoder to generate the output.
Although they are time-saving for fixing the parameters of PLMs, their performance is limited since they do not consider task-specific supervised information.
The fine-tuning-based methods employ PLMs as text encoders, where the generated representations are directly fed into the output layer. The parameters of the PLMs, along with task-specific parameters, are fine-tuned based on the task loss. These methods often require higher computing resources and are time-consuming, but they offer improved and promising performance compared to feature-based methods.
The fine-tuning-with-domain-adaption-based methods first conduct the domain-adaption for PLMs via continually training PLMs with designed tasks on the target data.
The adapted PLMs, along with task-specific layers, are then fine-tuned.
This domain adaptation process enables the PLMs to better capture domain-specific knowledge, resulting in improved performance compared to methods directly fine-tuning PLMs.
We will review and discuss these methods in more detail, as shown in Table {{table:d370596f-6676-49ca-85c0-b859bc48ad40}} .


Detection of sweet pepper and rock melon fruits

Identify obstacles in row crops and grass mowing

Detect obstacles that are distant, heavily occluded and unknown

Images of three fruit varieties: apples (726), almonds (385) and mangoes (1154)

Deep fruit detection in orchards

Abstract: An accurate and reliable image based fruit detection system is critical for supporting higher level agriculture tasks such as yield mapping and robotic harvesting. This paper presents the use of a state-of-the-art object detection framework, Faster R-CNN, in the context of fruit detection in orchards, including mangoes, almonds and apples. Ablation studies are presented to better understand the practical deployment of the detection network, including how much training data is required to capture variability in the dataset. Data augmentation techniques are shown to yield significant performance gains, resulting in a greater than two-fold reduction in the number of training images required. In contrast, transferring knowledge between orchards contributed to negligible performance gain over initialising the Deep Convolutional Neural Network directly from ImageNet features. Finally, to operate over orchard data containing between 100–1000 fruit per image, a tiling approach is introduced for the Faster R-CNN framework. The study has resulted in the best yet detection performance for these orchards relative to previous works, with an F1-score of > 0.9 achieved for apples and mangoes.

DeepFruits: A Fruit Detection System Using Deep Neural Networks

Abstract: This paper presents a novel approach to fruit detection using deep convolutional neural networks. The aim is to build an accurate, fast and reliable fruit detection system, which is a vital element of an autonomous agricultural robotic platform; it is a key element for fruit yield estimation and automated harvesting. Recent work in deep neural networks has led to the development of a state-of-the-art object detector termed Faster Region-based CNN (Faster R-CNN). We adapt this model, through transfer learning, for the task of fruit detection using imagery obtained from two modalities: colour (RGB) and Near-Infrared (NIR). Early and late fusion methods are explored for combining the multi-modal (RGB and NIR) information. This leads to a novel multi-modal Faster R-CNN model, which achieves state-of-the-art results compared to prior work with the F1 score, which takes into account both precision and recall performances improving from 0.807 to 0.838 for the detection of sweet pepper. In addition to improved accuracy, this approach is also much quicker to deploy for new fruits, as it requires bounding box annotation rather than pixel-level annotation (annotating bounding boxes is approximately an order of magnitude quicker to perform). The model is retrained to perform the detection of seven fruits, with the entire process taking four hours to annotate and train the new model per fruit.

Deep-plant: Plant identification with convolutional neural networks

Abstract: This paper studies convolutional neural networks (CNN) to learn unsupervised feature representations for 44 different plant species, collected at the Royal Botanic Gardens, Kew, England. To gain intuition on the chosen features from the CNN model (opposed to a `black box' solution), a visualisation technique based on the deconvolutional networks (DN) is utilized. It is found that venations of different order have been chosen to uniquely represent each of the plant species. Experimental results using these CNN features with different classifiers show consistency and superiority compared to the state-of-the art solutions which rely on hand-crafted features.

Using Deep Learning to Challenge Safety Standard for Highly Autonomous Machines in Agriculture

Abstract: In this paper, an algorithm for obstacle detection in agricultural fields is presented. The algorithm is based on an existing deep convolutional neural net, which is fine-tuned for detection of a specific obstacle. In ISO/DIS 18497, which is an emerging standard for safety of highly automated machinery in agriculture, a barrel-shaped obstacle is defined as the obstacle which should be robustly detected to comply with the standard. We show that our fine-tuned deep convolutional net is capable of detecting this obstacle with a precision of 99 . 9 % in row crops and 90 . 8 % in grass mowing, while simultaneously not detecting people and other very distinct obstacles in the image frame. As such, this short note argues that the obstacle defined in the emerging standard is not capable of ensuring safe operations when imaging sensors are part of the safety system.

Using Deep Learning for Image-Based Plant Disease Detection

Abstract: Crop diseases are a major threat to food security, but their rapid identification remains difficult in many parts of the world due to the lack of the necessary infrastructure. The combination of increasing global smartphone penetration and recent advances in computer vision made possible by deep learning has paved the way for smartphone-assisted disease diagnosis. Using a public dataset of 54,306 images of diseased and healthy plant leaves collected under controlled conditions, we train a deep convolutional neural network to identify 14 crop species and 26 diseases (or absence thereof). The trained model achieves an accuracy of 99.35% on a held-out test set, demonstrating the feasibility of this approach. Overall, the approach of training deep learning models on increasingly large and publicly available image datasets presents a clear path toward smartphone-assisted crop disease diagnosis on a massive global scale.

DeepAnomaly: Combining Background Subtraction and Deep Learning for Detecting Obstacles and Anomalies in an Agricultural Field

Abstract: Convolutional neural network (CNN)-based systems are increasingly used in autonomous vehicles for detecting obstacles. CNN-based object detection and per-pixel classification (semantic segmentation) algorithms are trained for detecting and classifying a predefined set of object types. These algorithms have difficulties in detecting distant and heavily occluded objects and are, by definition, not capable of detecting unknown object types or unusual scenarios. The visual characteristics of an agriculture field is homogeneous, and obstacles, like people, animals and other obstacles, occur rarely and are of distinct appearance compared to the field. This paper introduces DeepAnomaly, an algorithm combining deep learning and anomaly detection to exploit the homogenous characteristics of a field to perform anomaly detection. We demonstrate DeepAnomaly as a fast state-of-the-art detector for obstacles that are distant, heavily occluded and unknown. DeepAnomaly is compared to state-of-the-art obstacle detectors including “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks” (RCNN). In a human detector test case, we demonstrate that DeepAnomaly detects humans at longer ranges (45–90 m) than RCNN. RCNN has a similar performance at a short range (0–30 m). However, DeepAnomaly has much fewer model parameters and (182 ms/25 ms =) a 7.28-times faster processing time per image. Unlike most CNN-based methods, the high accuracy, the low computation time and the low memory footprint make it suitable for a real-time system running on a embedded GPU (Graphics Processing Unit).

Deep Neural Networks Based Recognition of Plant Diseases by Leaf Image Classification

Abstract: The latest generation of convolutional neural networks (CNNs) has achieved impressive results in the field of image classification. This paper is concerned with a new approach to the development of plant disease recognition model, based on leaf image classification, by the use of deep convolutional networks. Novel way of training and the methodology used facilitate a quick and easy system implementation in practice. The developed model is able to recognize 13 different types of plant diseases out of healthy leaves, with the ability to distinguish plant leaves from their surroundings. According to our knowledge, this method for plant disease recognition has been proposed for the first time. All essential steps required for implementing this disease recognition model are fully described throughout the paper, starting from gathering images in order to create a database, assessed by agricultural experts. Caffe, a deep learning framework developed by Berkley Vision and Learning Centre, was used to perform the deep CNN training. The experimental results on the developed model achieved precision between 91% and 98%, for separate class tests, on average 96.3%.

Deep Count: Fruit Counting Based on Deep Simulated Learning

Abstract: Recent years have witnessed significant advancement in computer vision research based on deep learning. Success of these tasks largely depends on the availability of a large amount of training samples. Labeling the training samples is an expensive process. In this paper, we present a simulated deep convolutional neural network for yield estimation. Knowing the exact number of fruits, flowers, and trees helps farmers to make better decisions on cultivation practices, plant disease prevention, and the size of harvest labor force. The current practice of yield estimation based on the manual counting of fruits or flowers by workers is a very time consuming and expensive process and it is not practical for big fields. Automatic yield estimation based on robotic agriculture provides a viable solution in this regard. Our network is trained entirely on synthetic data and tested on real data. To capture features on multiple scales, we used a modified version of the Inception-ResNet architecture. Our algorithm counts efficiently even if fruits are under shadow, occluded by foliage, branches, or if there is some degree of overlap amongst fruits. Experimental results show a 91% average test accuracy on real images and 93% on synthetic images.

Use of state-of-the-art CNN in Smart Farms

The table {{table:c755a2b7-27aa-45be-927b-0cb7a02bceb5}} , shows use of state-of-the-art CNN in agriculture and in particular the areas of plant and leaf disease detection, animal face identification, plant recognition, land cover classification, fruit counting and identification of weeds. It consist of 5 columns to show: the problem description, size of data used, accuracy according to the metrics used, the state-of-the-art CNN used and reference literature.