Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

VGGFace2: A Dataset for Recognising Faces across Pose and Age

Abstract: In this paper, we introduce a new large-scale face dataset named VGGFace2. The dataset contains 3.31 million images of 9131 subjects, with an average of 362.6 images for each subject. Images are downloaded from Google Image Search and have large variations in pose, age, illumination, ethnicity and profession (e.g. actors, athletes, politicians). The dataset was collected with three goals in mind: (i) to have both a large number of identities and also a large number of images for each identity; (ii) to cover a large range of pose, age and ethnicity; and (iii) to minimise the label noise. We describe how the dataset was collected, in particular the automated and manual filtering stages to ensure a high accuracy for the images of each identity. To assess face recognition performance using the new dataset, we train ResNet-50 (with and without Squeeze-and-Excitation blocks) Convolutional Neural Networks on VGGFace2, on MS-Celeb-1M, and on their union, and show that training on VGGFace2 leads to improved recognition performance over pose and age. Finally, using the models trained on these datasets, we demonstrate state-of-the-art performance on the IJB-A and IJB-B face recognition benchmarks, exceeding the previous state-of-the-art by a large margin. The dataset and models are publicly available.

Statistic of source and target datasets used in this paper.

In this section we conduct extensive studies on general discrimination and propose our method of soft fine-tuning.
The section is organized as follows: in Sec.REF  we demonstrate what is general discrimination and its importance in transfer learning. The method of keeping it is stated in Sec.REF . By adjusting weights of source and target domain loss functions, we introduce the proposed soft fine-tuning in Sec.REF , and
Sec.REF  figures the key factor of yielding better general discrimination.
All the studies in this section are implemented with the following settings:
we use three models of different sizes: MobileNetV2 (14M, {{cite:c1d405b}}), ResNet-50 (98M, {{cite:02d15c9}}) and InceptionV4 (164M, {{cite:59dd889}}).
Models are pre-trained on ImageNet {{cite:a38123d}} and used in Stanford-40 {{cite:02d822c}}.
The Stanford-40 {{cite:02d822c}} dataset contains 40 categories on human actions in still images, it is chosen since there are few categories about actions in ILSVRC 2012 {{cite:31f1730}} (more details can be found in Table.{{table:b3cd8554-5228-4a01-a948-ac5b7c60fc83}} ).
We choose “baseline network" in {{cite:cafc7a3}} in the experiment,
which receives both the whole image and the bounding box image, concatenates their features and obtains classification results.
In training stage, images are resized to 256/256/320 and randomly cropped to {{formula:d29f6d39-1d9d-4a48-a3e1-cf3c3800b8e5}}  for MobileNetV2/ResNet-50/InceptionV4.

First, we measure if a small-scale dataset, or a multi-task learning framework helps. We use a pre-trained model and fine-tune it simultaneously on both Stanford Dogs {{cite:56216d5}} and Stanford-40 {{cite:02d822c}} datasets. The Stanford Dogs {{cite:56216d5}} dataset is another small dataset, whose statistic is shown in Table.{{table:b3cd8554-5228-4a01-a948-ac5b7c60fc83}} .
In this trial, the loss function of Eqn.REF  can be rewritten as:
{{formula:4aa3b795-f3e4-41da-9418-077fbc833d1a}}


Scalable Active Learning for Object Detection

Abstract: Deep Neural Networks trained in a fully supervised fashion are the dominant technology in perception-based autonomous driving systems. While collecting large amounts of unlabeled data is already a major undertaking, only a subset of it can be labeled by humans due to the effort needed for high-quality annotation. Therefore, finding the right data to label has become a key challenge. Active learning is a powerful technique to improve data efficiency for supervised learning methods, as it aims at selecting the smallest possible training set to reach a required performance. We have built a scalable production system for active learning in the domain of autonomous driving. In this paper, we describe the resulting high-level design, sketch some of the challenges and their solutions, present our current results at scale, and briefly describe the open problems and future directions.

Learning Loss for Active Learning

Abstract: The performance of deep neural networks improves with more annotated data. The problem is that the budget for annotation is limited. One solution to this is active learning, where a model asks human to annotate data that it perceived as uncertain. A variety of recent methods have been proposed to apply active learning to deep networks but most of them are either designed specific for their target tasks or computationally inefficient for large networks. In this paper, we propose a novel active learning method that is simple but task-agnostic, and works efficiently with the deep networks. We attach a small parametric module, named ``loss prediction module,'' to a target network, and learn it to predict target losses of unlabeled inputs. Then, this module can suggest data that the target model is likely to produce a wrong prediction. This method is task-agnostic as networks are learned from a single loss regardless of target tasks. We rigorously validate our method through image classification, object detection, and human pose estimation, with the recent network architectures. The results demonstrate that our method consistently outperforms the previous methods over the tasks.

Active Learning for Deep Object Detection via Probabilistic Modeling

Abstract: Active learning aims to reduce labeling costs by selecting only the most informative samples on a dataset. Few existing works have addressed active learning for object detection. Most of these methods are based on multiple models or are straightforward extensions of classification methods, hence estimate an image’s informativeness using only the classification head. In this paper, we propose a novel deep active learning approach for object detection. Our approach relies on mixture density networks that estimate a probabilistic distribution for each localization and classification head’s output. We explicitly estimate the aleatoric and epistemic uncertainty in a single forward pass of a single model. Our method uses a scoring function that aggregates these two types of uncertainties for both heads to obtain every image’s informativeness score. We demonstrate the efficacy of our approach in PASCAL VOC and MS-COCO datasets. Our approach outperforms single-model based methods and performs on par with multi-model based methods at a fraction of the computing cost. Code is available at https://github.com/NVlabs/AL-MDN.

Deep Active Learning for Efficient Training of a LiDAR 3D Object Detector

Abstract: Training a deep object detector for autonomous driving requires a huge amount of labeled data. While recording data via on-board sensors such as camera or LiDAR is relatively easy, annotating data is very tedious and time-consuming, especially when dealing with 3D LiDAR points or radar data. Active learning has the potential to minimize human annotation efforts while maximizing the object detector's performance. In this work, we propose an active learning method to train a LiDAR 3D object detector with the least amount of labeled training data necessary. The detector leverages 2D region proposals generated from the RGB images to reduce the search space of objects and speed up the learning process. Experiments show that our proposed method works under different uncertainty estimations and query functions, and can save up to 60% of the labeling efforts while reaching the same network performance.

Abstract: The great success that deep models have achieved in the past is mainly owed to large amounts of labeled training data. However, the acquisition of labeled data for new tasks aside from existing benchmarks is both challenging and costly. Active learning can make the process of labeling new data more efficient by selecting unlabeled samples which, when labeled, are expected to improve the model the most. In this paper, we combine a novel method of active learning for object detection with an incremental learning scheme to enable continuous exploration of new unlabeled datasets. We propose a set of uncertainty-based active learning metrics suitable for most object detectors. Furthermore, we present an approach to leverage class imbalances during sample selection. All methods are evaluated systematically in a continuous exploration context on the PASCAL VOC 2012 dataset.

Consistency-based Active Learning for Object Detection

Abstract: Active learning aims to improve the performance of the task model by selecting the most informative samples with a limited budget. Unlike most recent works that focus on applying active learning for image classification, we propose an effective Consistency-based Active Learning method for object Detection (CALD), which fully explores the consistency between the original and augmented data. CALD has three appealing benefits. (i) CALD is systematically designed by investigating the weaknesses of existing active learning methods, which do not take the unique challenges of object detection into account. (ii) CALD unifies box regression and classification with a single metric, which is not concerned with active learning methods for classification. CALD also focuses on the most informative local region rather than the whole image, which is beneficial for object detection. (iii) CALD not only gauges individual information for sample selection but also leverages mutual information to encourage a balanced data distribution. Extensive experiments show that CALD significantly outperforms existing state-of-the-art task-agnostic and detection-specific active learning methods on general object detection datasets. Based on the Faster R-CNN detector, CALD consistently surpasses the baseline method (random selection) by 2.9/2.8/0.8 mAP on average on PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO. Code is available at https://github.com/we1pingyu/CALD

Multiple Instance Active Learning for Object Detection

Abstract: Despite the substantial progress of active learning for image recognition, there still lacks an instance-level active learning method specified for object detection. In this paper, we propose Multiple Instance Active Object Detection (MI-AOD), to select the most informative images for detector training by observing instance-level uncertainty. MI-AOD defines an instance uncertainty learning module, which leverages the discrepancy of two adversarial instance classifiers trained on the labeled set to predict instance uncertainty of the unlabeled set. MI-AOD treats unlabeled images as instance bags and feature anchors in images as instances, and estimates the image uncertainty by re-weighting instances in a multiple instance learning (MIL) fashion. Iterative instance uncertainty learning and re-weighting facilitate suppressing noisy instances, toward bridging the gap between instance uncertainty and image-level uncertainty. Experiments validate that MI-AOD sets a solid baseline for instance-level active learning. On commonly used object detection datasets, MI-AOD outperforms state-of-the-art methods with significant margins, particularly when the labeled sets are small. Code is available at https://github.com/yuantn/MI-AOD.

ALBench: A Framework for Evaluating Active Learning in Object Detection

Abstract: Active learning is an important technology for automated machine learning systems. In contrast to Neural Architecture Search (NAS) which aims at automating neural network architecture design, active learning aims at automating training data selection. It is especially critical for training a long-tailed task, in which positive samples are sparsely distributed. Active learning alleviates the expensive data annotation issue through incrementally training models powered with efficient data selection. Instead of annotating all unlabeled samples, it iteratively selects and annotates the most valuable samples. Active learning has been popular in image classification, but has not been fully explored in object detection. Most of current approaches on object detection are evaluated with different settings, making it difficult to fairly compare their performance. To facilitate the research in this field, this paper contributes an active learning benchmark framework named as ALBench for evaluating active learning in object detection. Developed on an automatic deep model training system, this ALBench framework is easy-to-use, compatible with different active learning algorithms, and ensures the same training and testing protocols. We hope this automated benchmark system help researchers to easily reproduce literature's performance and have objective comparisons with prior arts. The code will be release through Github.

Training and testing setups utilized in existing active learning works for object detection, where “Added Size” denotes the number of selected samples at each iteration. The training stops when the size of augmented training set reaches the “Stop Size”. It shows that current methods employ various benchmark setups which prevents objective comparison among them.

Research on active learning for object detection has not been as popular as that for image classification. This is partially because it involves more training setups than traditional learning strategies, making the algorithm implementation, repeat, and comparison difficult. Besides that, existing active learning algorithms for detection follow different evaluation paradigms, making their performance not directly comparable. We summarize the training and test setups in existing works in Table {{table:686f8a9d-bc0e-4ff7-a1da-f31427182667}} . The listed studies could use:

Background: Active Learning for Object Detection

A fair comparison among those algorithms is critical for future studies. As illustrated in Table {{table:686f8a9d-bc0e-4ff7-a1da-f31427182667}} , current works apply different setups, making a fair comparison difficult. One of related works {{cite:74b9180}} focuses on the pedestrian detection task, and is not evaluated on general object detection datasets like COCO {{cite:81e1dbd}} and VOC {{cite:464ac9c}}. Similarly, {{cite:8919145}} focuses on the object detection in autonomous driving scenarios. Existing approaches also select different numbers of samples at each iteration. For instance, {{cite:d7ea46e}} uses 100k images as the initial training set, and selects 200k images at each iteration. Another work {{cite:2eb56b0}} trains the YOLO {{cite:da9bd69}} detector and considers the category imbalance issue during training. It only selects 50 images at each iteration. Annotating 200k images at each iteration is expensive for real applications. Selecting too few samples degrades the speed of performance enhancement, leading to a time-consuming training process. Therefore, it not reasonable to select either too many or too few samples for active learning.


Uses a database of the intermediate ResNet-50 features to find similar studies

Generate reports by classifying CXRs, and finding and modifying similar reports

Extracts features from Chest X-rays and uses another network to write reports.

Generates radiology reports by training on classification labels and report text

Mahalanobis distance on network layers to detect out-of-distribution samples

Comprehensive Computer-Aided Decision Support Framework to Diagnose Tuberculosis From Chest X-Ray Images: Data Mining Study

Abstract: Background Tuberculosis (TB) is one of the most infectious diseases that can be fatal. Its early diagnosis and treatment can significantly reduce the mortality rate. In the literature, several computer-aided diagnosis (CAD) tools have been proposed for the efficient diagnosis of TB from chest radiograph (CXR) images. However, the majority of previous studies adopted conventional handcrafted feature-based algorithms. In addition, some recent CAD tools utilized the strength of deep learning methods to further enhance diagnostic performance. Nevertheless, all these existing methods can only classify a given CXR image into binary class (either TB positive or TB negative) without providing further descriptive information. Objective The main objective of this study is to propose a comprehensive CAD framework for the effective diagnosis of TB by providing visual as well as descriptive information from the previous patients’ database. Methods To accomplish our objective, first we propose a fusion-based deep classification network for the CAD decision that exhibits promising performance over the various state-of-the-art methods. Furthermore, a multilevel similarity measure algorithm is devised based on multiscale information fusion to retrieve the best-matched cases from the previous database. Results The performance of the framework was evaluated based on 2 well-known CXR data sets made available by the US National Library of Medicine and the National Institutes of Health. Our classification model exhibited the best diagnostic performance (0.929, 0.937, 0.921, 0.928, and 0.965 for F1 score, average precision, average recall, accuracy, and area under the curve, respectively) and outperforms the performance of various state-of-the-art methods. Conclusions This paper presents a comprehensive CAD framework to diagnose TB from CXR images by retrieving the relevant cases and their clinical observations from the previous patients’ database. These retrieval results assist the radiologist in making an effective diagnostic decision related to the current medical condition of a patient. Moreover, the retrieval results can facilitate the radiologists in subjectively validating the CAD decision.

Chest X-ray Report Generation through Fine-Grained Label Learning

Abstract: Obtaining automated preliminary read reports for common exams such as chest X-rays will expedite clinical workflows and improve operational efficiencies in hospitals. However, the quality of reports generated by current automated approaches is not yet clinically acceptable as they cannot ensure the correct detection of a broad spectrum of radiographic findings nor describe them accurately in terms of laterality, anatomical location, severity, etc. In this work, we present a domain-aware automatic chest X-ray radiology report generation algorithm that learns fine-grained description of findings from images and uses their pattern of occurrences to retrieve and customize similar reports from a large report database. We also develop an automatic labeling algorithm for assigning such descriptors to images and build a novel deep learning network that recognizes both coarse and fine-grained descriptions of findings. The resulting report generation algorithm significantly outperforms the state of the art using established score metrics.

Vispi: Automatic Visual Perception and Interpretation of Chest X-rays

Abstract: Medical imaging contains the essential information for rendering diagnostic and treatment decisions. Inspecting (visual perception) and interpreting image to generate a report are tedious clinical routines for a radiologist where automation is expected to greatly reduce the workload. Despite rapid development of natural image captioning, computer-aided medical image visual perception and interpretation remain a challenging task, largely due to the lack of high-quality annotated image-report pairs and tailor-made generative models for sufficient extraction and exploitation of localized semantic features, particularly those associated with abnormalities. To tackle these challenges, we present Vispi, an automatic medical image interpretation system, which first annotates an image via classifying and localizing common thoracic diseases with visual support and then followed by report generation from an attentive LSTM model. Analyzing an open IU X-ray dataset, we demonstrate a superior performance of Vispi in disease classification, localization and report generation using automatic performance evaluation metrics ROUGE and CIDEr.

Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment

Abstract: Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limited. To cope with these practical challenges, we propose a generative encoder-decoder model and focus on chest x-ray images and reports with the following improvements. First, we pretrain the encoder with a large number of chest x-ray images to accurately recognize 14 common radiographic observations, while taking advantage of the multi-view images by enforcing the cross-view consistency. Second, we synthesize multi-view visual features based on a sentence-level attention mechanism in a late fusion fashion. In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images. Such concepts are fused with each decoding step by a word-level attention model. The experimental results conducted on the Indiana University Chest X-Ray dataset demonstrate that the proposed model achieves the state-of-the-art performance compared with other baseline approaches.

FRODO: Free rejection of out-of-distribution samples: application to chest x-ray analysis

Abstract: In this work, we propose a method to reject out-of-distribution samples which can be adapted to any network architecture and requires no additional training data. Publicly available chest x-ray data (38,353 images) is used to train a standard ResNet-50 model to detect emphysema. Feature activations of intermediate layers are used as descriptors defining the training data distribution. A novel metric, FRODO, is measured by using the Mahalanobis distance of a new test sample to the training data distribution. The method is tested using a held-out test dataset of 21,176 chest x-rays (in-distribution) and a set of 14,821 out-of-distribution x-ray images of incorrect orientation or anatomy. In classifying test samples as in or out-of distribution, our method achieves an AUC score of 0.99.

Other Studies (Section ).Tasks: IL=Image-level Predictions, IR=Image Retrieval, OD=Out-of-Distribution, RE=Registration, RG=Report Generation, RP=Report Parsing. Bold font in tasks implies that this additional task is central to the work and the study also appears in another table in this paper.Labels: C=ChestX-Ray14, H=Heart, L=Lung, Q=Image Quality, T=Triage/Abnormal, TB=Tuberculosis, X=CheXpert, Z=Other.Datasets: C=ChestX-ray14, J=JSRT+SCR, M=MIMIC-CXR, MO=Montgomery, O=Open-i, PR=Private, S=Shenzen, X=CheXpert.

In this section we detail each public dataset that is encountered in the literature included in this review as well as any others available to the best of our knowledge. Details are provided in Table {{table:69705ff8-9368-426e-b430-b62d1c5af9cf}} . Each dataset is given an acronym which is used in the literature review tables (Tables  {{table:e2ad4a3d-50a8-4d08-9026-9b14e4c65afb}}  to  {{table:e36fbf84-8f24-4a95-aa9a-69a76dfe0fa3}} ) to indicate that the dataset was used in the specified work.

In this section we review articles with a primary application that does not fit into any of the categories detailed in Sections REF  to REF  (14 studies). These works are detailed fully in Table {{table:e36fbf84-8f24-4a95-aa9a-69a76dfe0fa3}} .


Multiple multiple-antenna eavesdroppers, imperfect SIC

a single-antenna eavesdropper, imperfect SIC

Multi-Objective Optimization for Robust Power Efficient and Secure Full-Duplex Wireless Communication Systems

Abstract: In this paper, we investigate the power efficient resource allocation algorithm design for secure multiuser wireless communication systems employing a full-duplex (FD) base station (BS) for serving multiple half-duplex (HD) downlink (DL) and uplink (UL) users simultaneously. We propose a multi-objective optimization framework to study two conflicting yet desirable design objectives, i.e., total DL transmit power minimization and total UL transmit power minimization. To this end, the weighed Tchebycheff method is adopted to formulate the resource allocation algorithm design as a multi-objective optimization problem (MOOP). The considered MOOP takes into account the quality-of-service requirements of all legitimate users for guaranteeing secure DL and UL transmission in the presence of potential eavesdroppers. Thereby, secure UL transmission is enabled by the FD BS and would not be possible with an HD BS. The imperfectness of the channel state information of the eavesdropping channels and the inter-user interference channels is incorporated for robust resource allocation algorithm design. Although the considered MOOP is non-convex, we solve it optimally by semidefinite programming relaxation. Simulation results not only unveil the trade-off between the total DL transmit power and the total UL transmit power, but also confirm the robustness of the proposed algorithm against potential eavesdroppers.

Transceiver Design to Maximize Sum Secrecy Rate in Full Duplex SWIPT Systems

Abstract: This letter considers secrecy simultaneous wireless information and power transfer (SWIPT) in full duplex systems. In such a system, full duplex capable base station (FD-BS) is designed to transmit data to one downlink user and concurrently receive data from one uplink user, while one idle user harvests the radio-frequency (RF) signals energy to extend its lifetime. Moreover, to prevent eavesdropping, artificial noise (AN) is exploited by FD-BS to degrade the channel of the idle user, as well as to provide energy supply to the idle user. To maximize the sum of downlink secrecy rate and uplink secrecy rate, we jointly optimize the information covariance matrix, AN covariance matrix and receiver vector, under the constraints of the sum transmission power of FD-BS and the minimum harvested energy of the idle user. Since the problem is non-convex, the log-exponential reformulation and sequential parametric convex approximation (SPCA) method are used. Extensive simulation results are provided and demonstrate that our proposed full duplex scheme extremely outperforms the half duplex scheme.

Secure Communications with Full Duplex Base Station

Considering a multiple-antenna full duplex base station which communicates with
a single-antenna transmitter and a single-antenna receiver simultaneously with single-antenna eavesdropper, F. Zhu et al.
investigate the joint precoding and AN generation design at the base station with global perfect CSI and perfect SIC
to guarantee both the uplink and downlink transmission security {{cite:7838a87}}.
This work is further extended to the imperfect SIC case {{cite:009d87c}}.
Y. Sun et al. study a more general system where a multiple-antenna full duplex base station receives information from
multiple single-antenna uplink users and transmit information to multiple single-antenna
downlink users simultaneously in the presence
of multiple potential eavesdroppers {{cite:f41633c}}, as shown in Fig. {{figure:ff8b70ec-4892-463d-94f0-1fe4581ef69d}} .
It is assumed that only imperfect CSIs of eavesdroppers are available at the
base station. A robust resource allocation scheme is designed to minimize
a total of uplink and downlink transmit power subject to the uplink and the downlink
rate and security rate constraints. As illustrated in Fig. {{figure:3204e180-42f0-4d08-b995-34e1b584c90a}} ,
the proposed design achieves significantly higher power efficiency comparing to the baseline ZFBF scheme.
Y. Wang et al. investigate the secure transmission for simultaneous wireless
information and power transfer full duplex base station systems {{cite:c1337cc}}.
A brief summary of above work is given in Table {{table:820f0e21-6c1e-4529-acd2-ec4d2b83c399}} .


CP-ABE based multi-authority data access control scheme in Fog-cloud computing systems.

Outsourced encryption and decryption computations.

A hybrid and fine-grained access control solution.

Most of the decryption process can be outsourced.

Secure and suitable in the Fog computing environment.

Perfectly applicable for resource-constrained IoT devices and applications.

User and attribute revocation can be performed efficiently.

Secure and highly efficient scheme.

Efficiency of data access is improved.

Key management cost is greatly reduced.

The limitation and drawbacks of this method is it can be applied only in centralized architecture.

A Secure and Verifiable Outsourced Access Control Scheme in Fog-Cloud Computing

Abstract: With the rapid development of big data and Internet of things (IOT), the number of networking devices and data volume are increasing dramatically. Fog computing, which extends cloud computing to the edge of the network can effectively solve the bottleneck problems of data transmission and data storage. However, security and privacy challenges are also arising in the fog-cloud computing environment. Ciphertext-policy attribute-based encryption (CP-ABE) can be adopted to realize data access control in fog-cloud computing systems. In this paper, we propose a verifiable outsourced multi-authority access control scheme, named VO-MAACS. In our construction, most encryption and decryption computations are outsourced to fog devices and the computation results can be verified by using our verification method. Meanwhile, to address the revocation issue, we design an efficient user and attribute revocation method for it. Finally, analysis and simulation results show that our scheme is both secure and highly efficient.

A Hybrid Scheme for Fine-Grained Search and Access Authorization in Fog Computing Environment

Abstract: In the fog computing environment, the encrypted sensitive data may be transferred to multiple fog nodes on the edge of a network for low latency; thus, fog nodes need to implement a search over encrypted data as a cloud server. Since the fog nodes tend to provide service for IoT applications often running on resource-constrained end devices, it is necessary to design lightweight solutions. At present, there is little research on this issue. In this paper, we propose a fine-grained owner-forced data search and access authorization scheme spanning user-fog-cloud for resource constrained end users. Compared to existing schemes only supporting either index encryption with search ability or data encryption with fine-grained access control ability, the proposed hybrid scheme supports both abilities simultaneously, and index ciphertext and data ciphertext are constructed based on a single ciphertext-policy attribute based encryption (CP-ABE) primitive and share the same key pair, thus the data access efficiency is significantly improved and the cost of key management is greatly reduced. Moreover, in the proposed scheme, the resource constrained end devices are allowed to rapidly assemble ciphertexts online and securely outsource most of decryption task to fog nodes, and mediated encryption mechanism is also adopted to achieve instantaneous user revocation instead of re-encrypting ciphertexts with many copies in many fog nodes. The security and the performance analysis show that our scheme is suitable for a fog computing environment.

The summary of Access Control Issue in the Fog environment across major survey papers

In accordance with the above study, and based on different access control method, it has been summarized into Table {{table:31c80ad2-a4ce-4d17-a8fe-80a14c6c8fcc}} .


Bioluminescent Swarm Optimization Algorithm

Abstract: Evolutionary Computation (EC) is a research area of metaheuristics mainly applied to real-world optimization problems. EC is inspired by biological mechanisms such as reproduction, mutation, recombination, natural selection and collective animal behavior. Two branches of EC can be highlighted: Evolutionary Algorithms (EA) comprising Genetic Algorithms (Goldberg, 1989), Genetic Programming (Koza, 1992), Differential Evolution (Storn & Price, 1997), Harmony Search (Geem et al., 2001), and others; and Swarm Intelligence (SI) comprising Ant Colony Optimization (ACO) (Dorigo & Stutzle, 2004) and Particle Swarm Optimization (PSO) (Kennedy & Eberhart, 2001; Poli et al., 2007) and others. The ACO metaheuristic1 is inspired by the foraging behavior of ants. On the other hand, the PSO metaheuristic2 is motivated by the coordinated movement of fish schools and bird flocks. Both ACO and PSO approaches have been applied successfully in a vast range of problems (Clerc, 2006; Dorigo & Stutzle, 2004). In recent years, new SI algorithms were proposed. They have in common biological inspirations, such as bacterial foraging (Passino, 2002), slime molds life cycle (Monismith & Mayfield, 2008), various bees behaviors (Karaboga & Akay, 2009), cockroaches infestation (Havens et al., 2008), mosquitoes host-seeking (Feng et al., 2009), bats echolocation (Yang, 2010), and fireflies bioluminescense (Krishnanand & Ghose, 2005; 2009; Yang, 2009). This work proposes a new swarm-based evolutionary approach based on the bioluminescent behavior of fireflies, called Bioluminescent Swarm Optimization (BSO) algorithm. The BSO uses two basic characteristics of the Glow-worm Swarm Optimization (GSO) algorithm proposed by (Krishnanand & Ghose, 2005): the luciferin attractant, and the stochastic neighbor selection. However, BSO goes further introducing new features such as: stochastic adaptive step sizing, global optimum attraction, leader movement, and mass extinction. Besides, the proposed algorithm is hybridized with two local search techniques: local unimodal sampling and single-dimension perturbation. All these features makes BSO a powerful algorithm for hard optimization problems. Experiments were done to analyze the sensitivity of the BSO to control parameters. Later, extensive experiments were performed using several benchmark functions with high

BAS: Beetle Antennae Search Algorithm for Optimization Problems

Abstract: Meta-heuristic algorithms have become very popular because of powerful performance on the optimization problem. A new algorithm called beetle antennae search algorithm (BAS) is proposed in the paper inspired by the searching behavior of longhorn beetles. The BAS algorithm imitates the function of antennae and the random walking mechanism of beetles in nature, and then two main steps of detecting and searching are implemented. Finally, the algorithm is benchmarked on 2 well-known test functions, in which the numerical results validate the efficacy of the proposed BAS algorithm.

An Improved Cockroach Swarm Optimization

Abstract: Hunger component is introduced to the existing cockroach swarm optimization (CSO) algorithm to improve its searching ability and population diversity. The original CSO was modelled with three components: chase-swarming, dispersion, and ruthless; additional hunger component which is modelled using partial differential equation (PDE) method is included in this paper. An improved cockroach swarm optimization (ICSO) is proposed in this paper. The performance of the proposed algorithm is tested on well known benchmarks and compared with the existing CSO, modified cockroach swarm optimization (MCSO), roach infestation optimization RIO, and hungry roach infestation optimization (HRIO). The comparison results show clearly that the proposed algorithm outperforms the existing algorithms.

Firefly Algorithms for Multimodal Optimization

Abstract: Nature-inspired algorithms are among the most powerful algorithms for optimization. This paper intends to provide a detailed description of a new Firefly Algorithm (FA) for multimodal optimization applications. We will compare the proposed firefly algorithm with other metaheuristic algorithms such as particle swarm optimization (PSO). Simulations and results indicate that the proposed firefly algorithm is superior to existing metaheuristic algorithms. Finally we will discuss its applications and implications for further research.

Algorithms based on insects behavior (part 1).

Behavior of individual insects with social interaction considers several tasks like navigation, foraging or escape from predators. This behavior have been inspired from dragonflies ( {{cite:89ae2b1}}), approaching or hovering over food sources of fruitful flies ( {{cite:a26a8ef}}, {{cite:86f5680}}), worms foraging ( {{cite:2af7db1}}), foraging and mating of butterflies ( {{cite:53e8e13}}, {{cite:fc63c51}}, {{cite:5726721}}) or only focus on males butterflies to mate-location behavior ( {{cite:088e872}}). Long-horn beetle for searching food and avoid predators ( {{cite:3d8b4ce}}), bark beetles searching for food and a nest ( {{cite:d39e2f6}}), and foraging of seven spot ladybirds ( {{cite:f56a518}}). Attraction and flashing behaviors of fireflies ( {{cite:7d7c928}}) or lampyridaeis to produce light by bioluminescense for attraction of preys or mating ( {{cite:e68ca85}}).
Foraging and social interaction of spiders to establish the positions of preys based on detection of vibrations of the spider web ( {{cite:02b78b3}}, {{cite:d8ac810}}) or black widow spiders bizarre mating ( {{cite:501858d}}). Wasps interact and allocate the tasks required in nest ( {{cite:a6e3b91}},{{cite:e93f4f6}}). Only a collective behavior of aggregation has been represented in several algorithms, inspired on glowworms ( {{cite:7573eb6}}), grasshopper and locust swarm ( {{cite:a05d858}}, {{cite:96a2d28}}), cockroaches ( {{cite:3a33829}}, {{cite:2e48def}}). Migration made by monarch buttlerflies ( {{cite:3ec78a7}}) Attraction of females crickets by sound emitted by males chirping of wings ( {{cite:be95f5e}}) or mating in bumblebees ( {{cite:0329893}}) and mayflies ( {{cite:add2486}}). Different behaviors of Mosquitos have been used considering seeking a host ( {{cite:dae2dcd}}), selection of habitat lo lay eggs ( {{cite:e1a3118}}), and to find a hole in a net ( {{cite:0a8cbc4}}). All algorithms based on insects behavior are shown in Tables {{table:1f3781c9-714c-4024-afeb-8e3d3dcf3198}}  and {{table:dee6c73f-ec0a-43ce-8d64-1a2c22b97065}} .


BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Abstract: Abstract Motivation Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. Results We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. Availability and implementation We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.

BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition

Abstract: In recent years, with the growing amount of biomedical documents, coupled with advancement in natural language processing algorithms, the research on biomedical named entity recognition (BioNER) has increased exponentially. However, BioNER research is challenging as NER in the biomedical domain are: (i) often restricted due to limited amount of training data, (ii) an entity can refer to multiple types and concepts depending on its context and, (iii) heavy reliance on acronyms that are sub-domain specific. Existing BioNER approaches often neglect these issues and directly adopt the state-of-the-art (SOTA) models trained in general corpora, which often yields unsatisfactory results. We propose biomedical ALBERT (A Lite Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) - bioALBERT - an effective domain-specific pre-trained language model trained on a huge biomedical corpus designed to capture biomedical context-dependent NER. We adopted a self-supervised loss function used in ALBERT that targets modelling inter-sentence coherence to better learn context-dependent representations and incorporated parameter reduction strategies to minimise memory usage and enhance the training time in BioNER. In our experiments, BioALBERT outperformed comparative SOTA BioNER models on 8 biomedical NER benchmark datasets with 4 different entity types. The performance is increased for; (i) disease type corpora by 7.47% (NCBI- disease) and 10.63% (BC5CDR-disease); (ii) drug-chem type corpora by 4.61 % (BC5CDR-Chem) and 3.89% (BC4CHEMD); (iii) gene-protein type corpora by 12.25% (BC2GM) and 6.42% (JNLPBA); and (iv) species type corpora by 6.19% (LINNAEUS) and 23.71 % (Species-800) is observed which leads to a state-of-the-art results. The performance of a proposed model on four different biomedical entity types shows that our model is robust and generalisable in recognising biomedical entities in text.

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Abstract: Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly-available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition (NER). To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB.

Bio-Megatron: Larger Biomedical Domain Language Model

Abstract: There has been an influx of biomedical domain-specific language models, showing language models pre-trained on biomedical text perform better on biomedical domain benchmarks than those trained on general domain text corpora such as Wikipedia and Books. Yet, most works do not study the factors affecting each domain language application deeply. Additionally, the study of model size on domain-specific models has been mostly missing. We empirically study and evaluate several factors that can affect performance on domain language applications, such as the sub-word vocabulary set, model size, pre-training corpus, and domain transfer. We show consistent improvements on benchmarks with our larger BioMegatron model trained on a larger domain corpus, contributing to our understanding of domain language model applications. We demonstrate noticeable improvements over the previous state-of-the-art (SOTA) on standard biomedical NLP benchmarks of named entity recognition, relation extraction, and question answering. Model checkpoints and code are available at [this https URL] and [this https URL].

Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

Abstract: Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We make the datasets, pre-trained models, and codes publicly available at https://github.com/ ncbi-nlp/BLUE_Benchmark.

ELECTRAMed: a new pre-trained language representation model for biomedical NLP

Abstract: The overwhelming amount of biomedical scientific texts calls for the development of effective language models able to tackle a wide range of biomedical natural language processing (NLP) tasks. The most recent dominant approaches are domain-specific models, initialized with general-domain textual data and then trained on a variety of scientific corpora. However, it has been observed that for specialized domains in which large corpora exist, training a model from scratch with just in-domain knowledge may yield better results. Moreover, the increasing focus on the compute costs for pre-training recently led to the design of more efficient architectures, such as ELECTRA. In this paper, we propose a pre-trained domain-specific language model, called ELECTRAMed, suited for the biomedical field. The novel approach inherits the learning framework of the general-domain ELECTRA architecture, as well as its computational advantages. Experiments performed on benchmark datasets for several biomedical NLP tasks support the usefulness of ELECTRAMed, which sets the novel state-of-the-art result on the BC5CDR corpus for named entity recognition, and provides the best outcome in 2 over the 5 runs of the 7th BioASQ-factoid Challange for the question answering task.

Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT

Abstract: The availability of biomedical text data and advances in natural language processing (NLP) have made new applications in biomedical NLP possible. Language models trained or fine tuned using domain specific corpora can outperform general models, but work to date in biomedical NLP has been limited in terms of corpora and tasks. We present BioALBERT, a domain-specific adaptation of A Lite Bidirectional Encoder Representations from Transformers (ALBERT), trained on biomedical (PubMed and PubMed Central) and clinical (MIMIC-III) corpora and fine tuned for 6 different tasks across 20 benchmark datasets. Experiments show that BioALBERT outperforms the state of the art on named entity recognition (+11.09% BLURB score improvement), relation extraction (+0.80% BLURB score), sentence similarity (+1.05% BLURB score), document classification (+0.62% F1-score), and question answering (+2.83% BLURB score). It represents a new state of the art in 17 out of 20 benchmark datasets. By making BioALBERT models and data available, our aim is to help the biomedical NLP community avoid computational costs of training and establish a new set of baselines for future efforts across a broad range of biomedical NLP tasks.

Summary of scientific literature-based BPLMs. NER - Named Entity Recognition, RE - Relation Extraction, IR - Information Retrieval, QA - Question Answering.

As DSPT is expensive, most of the works developed in-domain T-PLMs by initializing from general BERT models and then further pretraining on biomedical text. BioBERT {{cite:6d5a663}} is the first biomedical pre-trained language model which is obtained by further pretraining general BERT on biomedical literature. BioBERTpt-bio {{cite:138c25a}} is obtained by further pretraining BERT Multilingual (base) on Brazilian biomedical corpus - scientific papers from PubMed (0.8M- only literature titles) + Scielo (health – 12.4M + biological- 3.2M: both titles and abstracts). BioMedBERT {{cite:8c6dd07}} is obtained by further pretraining BERT-large on BREATHE 1.0 corpus. BioMedBERT outperformed BioBERT on biomedical question answering. The key reason for the better performance of BioMedBERT is the diversity of biomedical text in the BREATHE corpus. The main drawback in developing biomedical models by further pretraining general models is the general vocabulary. To overcome this, researchers started to develop biomedical models by using DSPT. Microsoft researchers developed PubMed {{cite:7ba6ad8}} model by DSPT with in-domain vocabulary and whole word masking strategy. OuBioBERT {{cite:e1f5290}} is trained from scratch using focused PubMed abstracts (280M words) as core corpora and PubMed abstracts (2800M words) as satellite corpora. It outperforms BioBERT and BlueBERT in many of the tasks in the BLUE benchmark. Table {{table:029872f3-b3c6-4e82-bc44-9f4a102f32ef}}  contains summary of scientific literature-based T-BPLMs.


A Unified Framework for Planning in Adversarial and Cooperative Environments

Abstract: Users of AI systems may rely upon them to produce plans for achieving desired objectives. Such AI systems should be able to compute obfuscated plans whose execution in adversarial situations protects privacy, as well as legible plans which are easy for team members to understand in cooperative situations. We develop a unified framework that addresses these dual problems by computing plans with a desired level of comprehensibility from the point of view of a partially informed observer. For adversarial settings, our approach produces obfuscated plans with observations that are consistent with at least k goals from a set of decoy goals. By slightly varying our framework, we present an approach for goal legibility in cooperative settings which produces plans that achieve a goal while being consistent with at most j goals from a set of confounding goals. In addition, we show how the observability of the observer can be controlled to either obfuscate or clarify the next actions in a plan when the goal is known to the observer. We present theoretical results on the complexity analysis of our problems. We demonstrate the execution of obfuscated and legible plans in a cooking domain using a physical robot Fetch. We also provide an empirical evaluation to show the feasibility and usefulness of our approaches using IPC domains.

A Hierarchical Behavioral Dynamic Approach for Naturally Adaptive Human-Agent Pick-and-Place Interactions

Abstract: Interactive or collaborative pick-and-place tasks occur during all kinds of daily activities, for example, when two or more individuals pass plates, glasses, and utensils back and forth between each other when setting a dinner table or loading a dishwasher together. In the near future, participation in these collaborative pick-and-place tasks could also include robotic assistants. However, for human-machine and human-robot interactions, interactive pick-and-place tasks present a unique set of challenges. A key challenge is that high-level task-representational algorithms and preplanned action or motor programs quickly become intractable, even for simple interaction scenarios. Here we address this challenge by introducing a bioinspired behavioral dynamic model of free-flowing cooperative pick-and-place behaviors based on low-dimensional dynamical movement primitives and nonlinear action selection functions. Further, we demonstrate that this model can be successfully implemented as an artificial agent control architecture to produce effective and robust human-like behavior during human-agent interactions. Participants were unable to explicitly detect whether they were working with an artificial (model controlled) agent or another human-coactor, further illustrating the potential effectiveness of the proposed modeling approach for developing systems of robust real/embodied human-robot interaction more generally.

Overview of existing Legibility Frameworks

For each of these papers, we extracted some high-level information (properties), which is shown in tab. {{table:6892fd5b-ce0f-4e75-b799-8f9ec93f641a}} . In the table, the hyperparameters column denotes if the framework can be fine-tuned via hyper-parameters. The planning space column shows the space in which the optimization/learning takes place. Score denotes if the framework uses a cost-based (lower is better) or reward-based (higher is better) metric to evaluate the score of a trajectory; this column strongly correlates with the associated framework using path-planning or reinforcement learning. Goal space denotes if the framework creates a trajectory towards one (of many) discrete goals or towards a point in a continuous goal space. Finally, Eval Cost is a (rough) estimate of the computational complexity involved in creating a trajectory through a framework.

A potential explanation for this overall average agreement could be that we didn't fine-tune any hyperparameters. As shown in table {{table:6892fd5b-ce0f-4e75-b799-8f9ec93f641a}} , most frameworks introduce various hyper-parameters to adjust a framework to a specific use-case, e.g., they use weights for different parts of a reward function. To push the performance of those frameworks, a future study could use a hyper-parameter search combined with a bootstrapping-based evaluation assuming there are enough computational resources available to compensate the increased computational demand of such an approach.

The correlation negative of RL frameworks

A surprising finding was that all reward-based frameworks (see table {{table:6892fd5b-ce0f-4e75-b799-8f9ec93f641a}} ) show a negative correlation with the human baseline in Fig. {{figure:272d3ead-7c77-497d-b155-b37b249582ad}} .


Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

Datasets examined in transfer learning

We evaluated models on 12 image classification datasets ranging in training set size from 2,040 to 75,750 images (20 to 5,000 images per class; Table {{table:58ef0c0f-e4b8-4d06-b4ba-0bdad86544c5}} ). These datasets covered a wide range of image classification tasks, including superordinate-level object classification (CIFAR-10 {{cite:80975bb}}, CIFAR-100 {{cite:80975bb}}, PASCAL VOC 2007 {{cite:805dced}}, Caltech-101 {{cite:d36c81e}}); fine-grained object classification (Food-101 {{cite:3e19c29}}, Birdsnap {{cite:954f5d3}}, Stanford Cars {{cite:7dff88f}}, FGVC Aircraft {{cite:ad19817}}, Oxford-IIIT Pets {{cite:f466068}}); texture classification (DTD {{cite:2024429}}); and scene classification (SUN397 {{cite:b406726}}).


High-Resolution Image Synthesis with Latent Diffusion Models

Abstract: By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve a new state of the art for image inpainting and highly competitive performance on various tasks, including unconditional image generation, semantic scene synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs. Code is available at https://github.com/CompVis/latent-diffusion .

Adversarial Diffusion Distillation

Abstract: We introduce Adversarial Diffusion Distillation (ADD), a novel training approach that efficiently samples large-scale foundational image diffusion models in just 1-4 steps while maintaining high image quality. We use score distillation to leverage large-scale off-the-shelf image diffusion models as a teacher signal in combination with an adversarial loss to ensure high image fidelity even in the low-step regime of one or two sampling steps. Our analyses show that our model clearly outperforms existing few-step methods (GANs, Latent Consistency Models) in a single step and reaches the performance of state-of-the-art diffusion models (SDXL) in only four steps. ADD is the first method to unlock single-step, real-time image synthesis with foundation models. Code and weights available under https://github.com/Stability-AI/generative-models and https://huggingface.co/stabilityai/ .

Number of ground truth verbatims extracted from several models. For deep floyd and Midjourney, we use the top 500 prompts sorted from Eq.  from SDV1 (i.e. we don't perform the attack, just extract images). De-duplicated models seem less susceptible to exact verbatims, but are still vulnerable to template verbatim extractions. See Sec.

Template verbatims are more difficult to label as ground truth, not only because they require retrieval and masking, but also because they're not highly duplicated, as shown in Fig. {{figure:e1ac42b0-f27c-4fcc-b51d-e7ac3285506e}} . Note that Stable Diffusion v2 did deduplicated L2B before training. Unsurprisingly, we did not find any examples of exact verbatims being copied on Stable Diffusion v2, however, we still found many template verbatims, see Tab {{table:13e4c4ae-f92d-4063-a0a6-984b5cd9e442}} . Thus, a more relaxed duplication detection, such as the semantic duplicates found in SemDeDup {{cite:a696100}}, may be necessary to weed out these samples. We leave this for future work.

Having obtained our ground truth for Stable Diffusion V1 and V2, we test whether these prompts are also verbatim copied by a variety of other models, such as state of the art diffusion model DeepIF {{cite:d17dd66}}, and the closed source system MidJourney {{cite:9da0d7e}}. For Midjourney, this was done manually through discord for around 100 prompts, and we swept over the number of time steps until a verbatim was found. Tab. {{table:13e4c4ae-f92d-4063-a0a6-984b5cd9e442}}  shows the total number of ground truth verbatims extracted. With Midjourney {{cite:9da0d7e}}, the model is entirely black box, and likewise for the training set. Still, we find it still regurgitates some of the same prompts as other models, which are known to be trained on L2B. Interestingly, it is also less susceptible to the exact verbatim regurgitation like SDV2, so we hypothesize that Midjourney de-duplicated their training set before training. Finally, we note the popular stable diffusion checkpoint model openjourney, which was fine tuned from SDV1 using images and prompts from midjourney, and typically generates fantasy and surreal images better than the original model. In contrast, realistic vision, which is also a checkpoint, focuses on more photographic realism. Both models are corrupted and regurgitate the same prompts, and for realistic vision, the problem is exacerbated.


Securify: Practical Security Analysis of Smart Contracts

Abstract: Permissionless blockchains allow the execution of arbitrary programs (called smart contracts), enabling mutually untrusted entities to interact without relying on trusted third parties. Despite their potential, repeated security concerns have shaken the trust in handling billions of USD by smart contracts. To address this problem, we present Securify, a security analyzer for Ethereum smart contracts that is scalable, fully automated, and able to prove contract behaviors as safe/unsafe with respect to a given property. Securify's analysis consists of two steps. First, it symbolically analyzes the contract's dependency graph to extract precise semantic information from the code. Then, it checks compliance and violation patterns that capture sufficient conditions for proving if a property holds or not. To enable extensibility, all patterns are specified in a designated domain-specific language. Securify is publicly released, it has analyzed >18K contracts submitted by its users, and is regularly used to conduct security audits by experts. We present an extensive evaluation of Securify over real-world Ethereum smart contracts and demonstrate that it can effectively prove the correctness of smart contracts and discover critical violations.

ContractFuzzer: Fuzzing Smart Contracts for Vulnerability Detection

Abstract: Decentralized cryptocurrencies feature the use of blockchain to transfer values among peers on networks without central agency. Smart contracts are programs running on top of the blockchain consensus protocol to enable people make agreements while minimizing trusts. Millions of smart contracts have been deployed in various decentralized applications. The security vulnerabilities within those smart contracts pose significant threats to their applications. Indeed, many critical security vulnerabilities within smart contracts on Ethereum platform have caused huge financial losses to their users. In this work, we present ContractFuzzer, a novel fuzzer to test Ethereum smart contracts for security vulnerabilities. ContractFuzzer generates fuzzing inputs based on the ABI specifications of smart contracts, defines test oracles to detect security vulnerabilities, instruments the EVM to log smart contracts runtime behaviors, and analyzes these logs to report security vulnerabilities. Our fuzzing of 6991 smart contracts has flagged more than 459 vulnerabilities with high precision. In particular, our fuzzing tool successfully detects the vulnerability of the DAO contract that leads to $60 million loss and the vulnerabilities of Parity Wallet that have led to the loss of $30 million and the freezing of $150 million worth of Ether.

sFuzz: An Efficient Adaptive Fuzzer for Solidity Smart Contracts

Abstract: Smart contracts are Turing-complete programs that execute on the infrastructure of the blockchain, which often manage valuable digital assets. Solidity is one of the most popular programming languages for writing smart contracts on the Ethereum platform. Like traditional programs, smart contracts may contain vulnerabilities. Unlike traditional programs, smart contracts cannot be easily patched once they are deployed. It is thus important that smart contracts are tested thoroughly before deployment. In this work, we present an adaptive fuzzer for smart contracts on the Ethereum platform called sFuzz. Compared to existing Solidity fuzzers, sFuzz combines the strategy in the AFL fuzzer and an efficient lightweight multi-objective adaptive strategy targeting those hard-to-cover branches. sFuzz has been applied to more than 4 thousand smart contracts and the experimental results show that (1) sFuzz is efficient, e.g., two orders of magnitude faster than state-of-the-art tools; (2) sFuzz is effective in achieving high code coverage and discovering vulnerabilities; and (3) the different fuzzing strategies in sFuzz complement each other.

Sereum: Protecting Existing Smart Contracts Against Re-Entrancy Attacks

Abstract: Recently, a number of existing blockchain systems have witnessed major bugs and vulnerabilities within smart contracts. Although the literature features a number of proposals for securing smart contracts, these proposals mostly focus on proving the correctness or absence of a certain type of vulnerability within a contract, but cannot protect deployed (legacy) contracts from being exploited. In this paper, we address this problem in the context of re-entrancy exploits and propose a novel smart contract security technology, dubbed Sereum (Secure Ethereum), which protects existing, deployed contracts against re-entrancy attacks in a backwards compatible way based on run-time monitoring and validation. Sereum does neither require any modification nor any semantic knowledge of existing contracts. By means of implementation and evaluation using the Ethereum blockchain, we show that Sereum covers the actual execution flow of a smart contract to accurately detect and prevent attacks with a false positive rate as small as 0.06% and with negligible run-time overhead. As a by-product, we develop three advanced re-entrancy attacks to demonstrate the limitations of existing offline vulnerability analysis tools.

Harvey: A Greybox Fuzzer for Smart Contracts

Abstract: We present Harvey, an industrial greybox fuzzer for smart contracts, which are programs managing accounts on a blockchain. Greybox fuzzing is a lightweight test-generation approach that effectively detects bugs and security vulnerabilities. However, greybox fuzzers randomly mutate program inputs to exercise new paths; this makes it challenging to cover code that is guarded by narrow checks, which are satisfied by no more than a few input values. Moreover, most real-world smart contracts transition through many different states during their lifetime, e.g., for every bid in an auction. To explore these states and thereby detect deep vulnerabilities, a greybox fuzzer would need to generate sequences of contract transactions, e.g., by creating bids from multiple users, while at the same time keeping the search space and test suite tractable. In this experience paper, we explain how Harvey alleviates both challenges with two key fuzzing techniques and distill the main lessons learned. First, Harvey extends standard greybox fuzzing with a method for predicting new inputs that are more likely to cover new paths or reveal vulnerabilities in smart contracts. Second, it fuzzes transaction sequences in a targeted and demand-driven way. We have evaluated our approach on 27 real-world contracts. Our experiments show that the underlying techniques significantly increase Harvey's effectiveness in achieving high coverage and detecting vulnerabilities, in most cases orders-of-magnitude faster; they also reveal new insights about contract code.

VERISMART: A Highly Precise Safety Verifier for Ethereum Smart Contracts

Abstract: We present VERISMART, a highly precise verifier for ensuring arithmetic safety of Ethereum smart contracts. Writing safe smart contracts without unintended behavior is critically important because smart contracts are immutable and even a single flaw can cause huge financial damage. In particular, ensuring that arithmetic operations are safe is one of the most important and common security concerns of Ethereum smart contracts nowadays. In response, several safety analyzers have been proposed over the past few years, but state-of-the-art is still unsatisfactory; no existing tools achieve high precision and recall at the same time, inherently limited to producing annoying false alarms or missing critical bugs. By contrast, VERISMART aims for an uncompromising analyzer that performs exhaustive verification without compromising precision or scalability, thereby greatly reducing the burden of manually checking undiscovered or incorrectly-reported issues. To achieve this goal, we present a new domain-specific algorithm for verifying smart contracts, which is able to automatically discover and leverage transaction invariants that are essential for precisely analyzing smart contracts. Evaluation with real-world smart contracts shows that VERISMART can detect all arithmetic bugs with a negligible number of false alarms, far outperforming existing analyzers.

20 Tools published in top SE/Security conferences that can detect SWC weaknesses.

We totally find 20 tools published in top SE/Security conferences at the time we conducted our study. Table {{table:54ae9808-327d-4331-9534-df6e85951fdf}}  lists their names, the venues in which they were published, the SWC weaknesses that can be detected using these tools, and the dataset and evaluation method they use. The number of tools that can detect certain SWC weaknesses is also given in Figure {{figure:d2cf8c07-5595-4cc6-976d-95c33e5ceae9}} . From the figure, we can find that only 14 out of 37 SWC weaknesses could be detected using the current smart contract analysis tools. Reentrancy (SWC-107, 11 times) is the most popular weakness investigated in the academia, followed by Integer Overflow and Underflow (SWC-101, 7 times) and Block Values as a Proxy for Time (SWC-116, 7 times). Most of the SWC weaknesses are currently not supported by these tools.

The datasets and evaluation methods used in these tools are listed in the last column of Table {{table:54ae9808-327d-4331-9534-df6e85951fdf}} . There were four types of datasets (T1 to T4) and five kinds of methods (M1 to M5) used to evaluate tools, which are listed in Table {{table:a20eefad-c8ca-4b3c-b617-33ef59d8befd}} . Each evaluation method is represented in the form of “Method{Type(Size of Dataset)}". For example, M2{T1(1,524)} means that the dataset has 1,524 verified smart contracts (T1) T1 means the first type of the dataset. Then, the authors manually checked some/all the cases detected by the tool (M2). The symbol “+" means that multiple evaluations were conducted. For example, M3{T1(176)} + T1(47,398) means that a previous labeled dataset (M3), which consists of 176 verified smart contracts (T1), was reused to evaluate the tool. Then, the tool was used to analyze 47,398 smart contracts (T1) without checking their correctness.


EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Abstract: We introduce UCF101 which is currently the largest dataset of human actions. It consists of 101 action classes, over 13k clips and 27 hours of video data. The database consists of realistic user uploaded videos containing camera motion and cluttered background. Additionally, we provide baseline action recognition results on this new dataset using standard bag of words approach with overall performance of 44.5%. To the best of our knowledge, UCF101 is currently the most challenging dataset of actions due to its large number of classes, large number of clips and also unconstrained nature of such clips.

Describing Textures in the Wild

Natural Adversarial Examples

Abstract: We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets’ real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called IMAGENET-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called IMAGENET-O, which is the first out-of-distribution detection dataset created for ImageNet models. On IMAGENET-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on IMAGENET-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

Abstract: We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pre-trained with 1000× more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness.

Dataset details in our experiments.

The details of the 11 downstream datasets for few-show classification and 4 target datasets for domain generalization are shown in Table {{table:24b74289-a260-4dd8-8e62-1b9220e6b56e}} .


Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting

Abstract: The goal of traffic forecasting is to predict the future vital indicators (such as speed, volume and density) of the local traffic network in reasonable response time. Due to the dynamics and complexity of traffic network flow, typical simulation experiments and classic statistical methods cannot satisfy the requirements of mid-and-long term forecasting. In this work, we propose a novel deep learning framework, Spatio-Temporal Graph Convolutional Neural Network (ST-GCNN), to tackle this spatio-temporal sequence forecasting task. Instead of applying recurrent models to sequence learning, we build our model entirely on convolutional neural networks (CNNs) with gated linear units (GLU) and highway networks. The proposed architecture fully employs the graph structure of the road networks and enables faster training. Experiments show that our ST-GCNN network captures comprehensive spatio-temporal correlations throughout complex traffic network and consistently outperforms state-of-the-art baseline algorithms on several real-world traffic datasets.

Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks

Abstract: Modeling multivariate time series has long been a subject that has attracted researchers from a diverse range of fields including economics, finance, and traffic. A basic assumption behind multivariate time series forecasting is that its variables depend on one another but, upon looking closely, it is fair to say that existing methods fail to fully exploit latent spatial dependencies between pairs of variables. In recent years, meanwhile, graph neural networks (GNNs) have shown high capability in handling relational dependencies. GNNs require well-defined graph structures for information propagation which means they cannot be applied directly for multivariate time series where the dependencies are not known in advance. In this paper, we propose a general graph neural network framework designed specifically for multivariate time series data. Our approach automatically extracts the uni-directed relations among variables through a graph learning module, into which external knowledge like variable attributes can be easily integrated. A novel mix-hop propagation layer and a dilated inception layer are further proposed to capture the spatial and temporal dependencies within the time series. The graph learning, graph convolution, and temporal convolution modules are jointly learned in an end-to-end framework. Experimental results show that our proposed model outperforms the state-of-the-art baseline methods on 3 of 4 benchmark datasets and achieves on-par performance with other approaches on two traffic datasets which provide extra structural information.

Number of parameters and performance of baseline centralized models on PEMSD7M {{cite:85590a2}} dataset.

State-of-the-art multi-layer GNNs have been used to address the spatial-temporal nature of traffic prediction in centralized settings. In FL setting a challenge with GNN-based approaches is regarding vertical FL and in scenarios where each silo has a partial (overlapping) view of the graph. Many of the works in this domain are motivated by intelligent transportation systems often referred to as Road Side Units (RSU) where each unit has its own distribution of traffic patterns across the same city (Figure {{figure:0d27ef74-633c-435a-a9af-0b7fdc91ed84}} ).
One of the challenges with applying GNN based model for traffic forecasting is the large number of parameters that these models have and in the context of FL, the aggregation becomes a bottleneck for such models. For instance, while a simple Auto-Regressive Integrated Moving Average (ARIMA) model relies on a low number of parameters (order of 100s), the more complex models have an enormous number of parameters. Table {{table:8303a179-251c-4e2d-a9ec-4807b2abf121}}  shows the magnitude of parameters for baseline traffic prediction models over the PEMSD7M dataset {{cite:85590a2}}. The following works described below aim to tackle this problem by proposing alternative aggregation functions.