Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Abstract: We introduce UCF101 which is currently the largest dataset of human actions. It consists of 101 action classes, over 13k clips and 27 hours of video data. The database consists of realistic user uploaded videos containing camera motion and cluttered background. Additionally, we provide baseline action recognition results on this new dataset using standard bag of words approach with overall performance of 44.5%. To the best of our knowledge, UCF101 is currently the most challenging dataset of actions due to its large number of classes, large number of clips and also unconstrained nature of such clips.

Natural Adversarial Examples

Abstract: We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets’ real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called IMAGENET-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called IMAGENET-O, which is the first out-of-distribution detection dataset created for ImageNet models. On IMAGENET-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on IMAGENET-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

Abstract: We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pre-trained with 1000× more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness.

The detailed statistics of datasets used in experiments.

The datasets we used in the experiments follow CoOp {{cite:cdeeba8}}, which include 11 datasets for few-shot visual recognition and 4 ImageNet-based datasets for generalization (robustness) evaluation. The details of each dataset are shown in Table {{table:375d4976-21c6-49b5-96e9-ad499c5e39bc}} , including the number of classes, the sizes of training and testing sets, and original tasks.


DIML/CVL RGB-D Dataset: 2M RGB-D Images of Natural Indoor and Outdoor Scenes

Abstract: This manual is intended to provide a detailed description of the DIML/CVL RGB-D dataset. This dataset is comprised of 2M color images and their corresponding depth maps from a great variety of natural indoor and outdoor scenes. The indoor dataset was constructed using the Microsoft Kinect v2, while the outdoor dataset was built using the stereo cameras (ZED stereo camera and built-in stereo camera). Table I summarizes the details of our dataset, including acquisition, processing, format, and toolbox. Refer to Section II and III for more details.

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Abstract: A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available – current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval.

ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data

Abstract: Scene understanding is an active research area. Commercial depth sensors, such as Kinect, have enabled the release of several RGB-D datasets over the past few years which spawned novel methods in 3D scene understanding. More recently with the launch of the LiDAR sensor in Apple's iPads and iPhones, high quality RGB-D data is accessible to millions of people on a device they commonly use. This opens a whole new era in scene understanding for the Computer Vision community as well as app developers. The fundamental research in scene understanding together with the advances in machine learning can now impact people's everyday experiences. However, transforming these scene understanding methods to real-world experiences requires additional innovation and development. In this paper we introduce ARKitScenes. It is not only the first RGB-D dataset that is captured with a now widely available depth sensor, but to our best knowledge, it also is the largest indoor scene understanding data released. In addition to the raw and processed data from the mobile device, ARKitScenes includes high resolution depth maps captured using a stationary laser scanner, as well as manually labeled 3D oriented bounding boxes for a large taxonomy of furniture. We further analyze the usefulness of the data for two downstream tasks: 3D object detection and color-guided depth upsampling. We demonstrate that our dataset can help push the boundaries of existing state-of-the-art methods and it introduces new challenges that better represent real-world scenarios.

Comparison between current indoor datasets for depth estimation.

Our analysis of existing depth datasets (Table {{table:a2086d29-9a23-453a-978b-3a109036b9f6}} ) reveals that there are not a lot of datasets specifically captured for mobile AR. Therefore, researchers will not be able to evaluate their work effectively, and models will not be able to learn the special challenges and hard cases of mobile AR, preventing good generalization. We summarized a few popular indoor datasets in Table {{table:a2086d29-9a23-453a-978b-3a109036b9f6}} . Although various datasets exist for indoor depth estimation, most fail to address the specific challenges posed by mobile AR scenarios. For example, datasets such as NYU Depth V2, DLML Indoor, TUM RGB-D, and ScanNet are predominantly captured using specialized or static camera setups, thereby lacking the variability in camera parameters and environmental conditions typically encountered in mobile AR.


Argoverse: 3D Tracking and Forecasting With Rich Maps

Abstract: We present Argoverse, a dataset designed to support autonomous vehicle perception tasks including 3D tracking and motion forecasting. Argoverse includes sensor data collected by a fleet of autonomous vehicles in Pittsburgh and Miami as well as 3D tracking annotations, 300k extracted interesting vehicle trajectories, and rich semantic maps. The sensor data consists of 360 degree images from 7 cameras with overlapping fields of view, forward-facing stereo imagery, 3D point clouds from long range LiDAR, and 6-DOF pose. Our 290km of mapped lanes contain rich geometric and semantic metadata which are not currently available in any public dataset. All data is released under a Creative Commons license at Argoverse.org. In baseline experiments, we use map information such as lane direction, driveable area, and ground height to improve the accuracy of 3D object tracking. We use 3D object tracking to mine for more than 300k interesting vehicle trajectories to create a trajectory forecasting benchmark. Motion forecasting experiments ranging in complexity from classical methods (k-NN) to LSTMs demonstrate that using detailed vector maps with lane-level information substantially reduces prediction error. Our tracking and forecasting experiments represent only a superficial exploration of the potential of rich maps in robotic perception. We hope that Argoverse will enable the research community to explore these problems in greater depth.

3D Packing for Self-Supervised Monocular Depth Estimation

Abstract: Although cameras are ubiquitous, robotic platforms typically rely on active sensors like LiDAR for direct 3D perception. In this work, we propose a novel self-supervised monocular depth estimation method combining geometry with a new deep network, PackNet, learned only from unlabeled monocular videos. Our architecture leverages novel symmetrical packing and unpacking blocks to jointly learn to compress and decompress detail-preserving representations using 3D convolutions. Although self-supervised, our method outperforms other self, semi, and fully supervised methods on the KITTI benchmark. The 3D inductive bias in PackNet enables it to scale with input resolution and number of parameters without overfitting, generalizing better on out-of-domain data such as the NuScenes dataset. Furthermore, it does not require large-scale supervised pretraining on ImageNet and can run in real-time. Finally, we release DDAD (Dense Depth for Automated Driving), a new urban driving dataset with more challenging and accurate depth evaluation, thanks to longer-range and denser ground-truth depth generated from high-density LiDARs mounted on a fleet of self-driving cars operating world-wide.

We measured the average and minimum number of valid pixels on cropped images generated during one epoch of training. We observed that the KITTI, NYU-Depth V2, and Driving Stereo datasets provide dense ground-truth depth maps. However, the Argoverse and DDAD datasets provide significantly sparse depth maps with a valid pixel rate of approximately 1%, which can accidentally yield few valid pixels in the cropped image.

Second, the number of valid pixels is affected by the image crop. During the training of a monocular depth estimation model, several cropping methods, such as KB-crop, random-crop, and Garg-crop {{cite:e4d7081}}, are applied in a row to the image and depth map. Owing to the random-crop, the number of valid pixels contained in the ground-truth depth map varies every time. In a long training iteration, the cropped depth map can accidentally contain few valid pixels, such as {{formula:0644fbe5-5403-4322-b408-7220cf0bcc6a}}  or {{formula:8c7d6abf-f311-4c33-b3b8-75b28879ee2a}} , yielding a NaN loss when using an unbiased estimator of torch.var(x). For sparse datasets such as the Argoverse or DDAD datasets, this phenomenon occurs in real situations (Table {{table:3ec37048-9b4b-4bf2-b26b-96bccb989834}} ).

dataset contains RGB images and the corresponding ground truth depth maps. The ground truth depth map was measured using a LiDAR scanner that collected sparse 3D points; the other unmeasured areas remained empty. In the KITTI dataset, the maximum depth was {{formula:43b15532-c2ef-4644-a4d0-baed6ba6ae94}} . The KITTI dataset provides ground truth depth maps with an average valid pixel rate of approximately 20% (Table {{table:3ec37048-9b4b-4bf2-b26b-96bccb989834}} ). The size of the original image was {{formula:48428efb-0a58-4004-9084-1e89819d3b35}} , which was cropped by KB-crop, random-crop, and Garg-crop during training to obtain a {{formula:5f0feb12-a322-4386-9309-a566ee598853}}  image and a depth map.


This paper focuses on the interpretable classification of wire harness branches. The interpretability is visualized using a saliency map based on class activation mapping (CAM) {{cite:d12770e}}. The experimental results demonstrate the best classification based on a late prediction fusion {{cite:fd9fb1e}} of RGB and depth modalities, the deteriorating performance with the network pre-trained on the inpainting task, and the positive effect of elastic transform for data augmentation. The saliency maps promote the interpretability of the experimental results.

This paper explores the industrial bin-picking problem on wire harnesses and proposes learning a bin-picking policy to infer an optimal grasp and a post-grasping action based on a top-down depth image of the cluttered wire harnesses captured by a Photoneo PhoXi 3D scanner M. However, visual noise and heavy occlusion are two major challenges leading to failure results.

Tell Me, What Do You See?—Interpretable Classification of Wiring Harness Branches with Deep Neural Networks

Abstract: In the context of the robotisation of industrial operations related to manipulating deformable linear objects, there is a need for sophisticated machine vision systems, which could classify the wiring harness branches and provide information on where to put them in the assembly process. However, industrial applications require the interpretability of the machine learning system predictions, as the user wants to know the underlying reason for the decision made by the system. We propose several different neural network architectures that are tested on our novel dataset to address this issue. We conducted various experiments to assess the influence of modality, data fusion type, and the impact of data augmentation and pretraining. The outcome of the network is evaluated in terms of the performance and is also equipped with saliency maps, which allow the user to gain in-depth insight into the classifier’s operation, including a way of explaining the responses of the deep neural network and making system predictions interpretable by humans.

Learning Efficient Policies for Picking Entangled Wire Harnesses: An Approach to Industrial Bin Picking

Abstract: Wire harnesses are essential connecting components in manufacturing industry but are challenging to be automated in industrial tasks such as bin picking. They are long, flexible and tend to get entangled when randomly placed in a bin. This makes it difficult for the robot to grasp a single one in dense clutter. Besides, training or collecting data in simulation is challenging due to the difficulties in modeling the combination of deformable and rigid components for wire harnesses. In this work, instead of directly lifting wire harnesses, we propose to grasp and extract the target following a circle-like trajectory until it is untangled. We learn a policy from real-world data that can infer grasps and separation actions from visual observation. Our policy enables the robot to efficiently pick and separate entangled wire harnesses by maximizing success rates and reducing execution time. To evaluate our policy, we present a set of real-world experiments on picking wire harnesses. Our policy achieves an overall 84.6% success rate compared with 49.2% in baseline. We also evaluate the effectiveness of our policy under different clutter scenarios using unseen types of wire harnesses. Results suggest that our approach is feasible for handling wire harnesses in industrial bin picking.

Selected works and their contributions from the perspective of computer vision applications.

After searching and screening following the methodology described in Section , 14 articles were identified for analysis, including five articles and nine conference papers in terms of the document type, whose contributions from the perspective of computer vision applications are summarized in Table {{table:a84072e0-8f5c-4aa8-91fb-e1575bc5074a}} .
Figure {{figure:ccfb6d91-3ddf-47ac-88cc-8f1149ce905d}}  illustrates the development of research including computer vision applications in robotized assembly of wire harnesses throughout the past years regarding the total number of publications and citations, which indicates a long-term persistent effort on facilitating the robotized wire harness assembly with vision systems and increasing research in relevant subjects.
Figure {{figure:bff00e84-633a-4bce-8764-eb00bc4542aa}}  presents the citation relationships among the selected 14 studies, where each study is represented by a colorized circle in the 2D coordinate regarding the year of publication, the first author of the paper, and the component of wire harnesses the article focused on.
The citation relationships are visualized by solid and dashed arrows, representing the citation between two studies discussing the same and different wire harness components, respectively.
Figure {{figure:bff00e84-633a-4bce-8764-eb00bc4542aa}}  illustrates that the research on vision-enabled robotized wire harness assembly began with the recognition and manipulation of components of wire harnesses, such as clamps and connectors, and more recent studies have initiated new directions involving the understanding of the overall structure of wire harnesses.
However, the research regarding different components of wire harnesses ceased at different times.


Twitter100k: A Real-world Dataset for Weakly Supervised Cross-Media Retrieval

Abstract: This paper contributes a new large-scale dataset for weakly supervised cross-media retrieval, named Twitter100k. Current datasets, such as Wikipedia, NUS Wide and Flickr30k, have two major limitations. First, these datasets are lacking in content diversity, i.e., only some pre-defined classes are covered. Second, texts in these datasets are written in well-organized language, leading to inconsistency with realistic applications. To overcome these drawbacks, the proposed Twitter100k dataset is characterized by two aspects: 1) it has 100,000 image-text pairs randomly crawled from Twitter and thus has no constraint in the image categories; 2) text in Twitter100k is written in informal language by the users. Since strongly supervised methods leverage the class labels that may be missing in practice, this paper focuses on weakly supervised learning for cross-media retrieval, in which only text-image pairs are exploited during training. We extensively benchmark the performance of four subspace learning methods and three variants of the Correspondence AutoEncoder, along with various text features on Wikipedia, Flickr30k and Twitter100k. Novel insights are provided. As a minor contribution, inspired by the characteristic of Twitter100k, we propose an OCR-based cross-media retrieval method. In experiment, we show that the proposed OCR-based method improves the baseline performance.

An Overview of Cross-media Retrieval: Concepts, Methodologies, Benchmarks and Challenges

Abstract: Multimedia retrieval plays an indispensable role in big data utilization. Past efforts mainly focused on single-media retrieval. However, the requirements of users are highly flexible, such as retrieving the relevant audio clips with one query of image. So challenges stemming from the "media gap", which means that representations of different media types are inconsistent, have attracted increasing attention. Cross-media retrieval is designed for the scenarios where the queries and retrieval results are of different media types. As a relatively new research topic, its concepts, methodologies and benchmarks are still not clear in the literatures. To address these issues, we review more than 100 references, give an overview including the concepts, methodologies, major challenges and open issues, as well as build up the benchmarks including datasets and experimental results. Researchers can directly adopt the benchmarks to promptly evaluate their proposed methods. This will help them to focus on algorithm design, rather than the time-consuming compared methods and results. It is noted that we have constructed a new dataset XMedia, which is the first publicly available dataset with up to five media types (text, image, video, audio and 3D model). We believe this overview will attract more researchers to focus on cross-media retrieval and be helpful to them.

A summary of datasets in cross-media retrieval. For each dataset we identify the modality used to tackle the problem of cross-media retrieval.

Dataset plays a critical role in the evaluation of learning algorithm. Its selection is very important for feature extraction and training of different DL algorithms. We summarized some of the well-known cross-media datasets below, and Table {{table:929e5c1b-6eb7-458e-ac48-adf6c249a1f4}}  depicts a comparison evaluation among them.


Abdominal organ segmentation (overlap with KiTS, MSD)

Abdominal organs (liver, kidneys, spleen)

4-chamber (A4C) and apical 2-chamber (A2C) left wall

Large-scale collection of 10 Medical Segmentation Datasets

AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?

Abstract: With the unprecedented developments in deep learning, automatic segmentation of main abdominal organs seems to be a solved problem as state-of-the-art (SOTA) methods have achieved comparable results with inter-rater variability on many benchmark datasets. However, most of the existing abdominal datasets only contain single-center, single-phase, single-vendor, or single-disease cases, and it is unclear whether the excellent performance can generalize on diverse datasets. This paper presents a large and diverse abdominal CT organ segmentation dataset, termed AbdomenCT-1K, with more than 1000 (1K) CT scans from 12 medical centers, including multi-phase, multi-vendor, and multi-disease cases. Furthermore, we conduct a large-scale study for liver, kidney, spleen, and pancreas segmentation and reveal the unsolved segmentation problems of the SOTA methods, such as the limited generalization ability on distinct medical centers, phases, and unseen diseases. To advance the unsolved problems, we further build four organ segmentation benchmarks for fully supervised, semi-supervised, weakly supervised, and continual learning, which are currently challenging and active research topics. Accordingly, we develop a simple and effective method for each benchmark, which can be used as out-of-the-box methods and strong baselines. We believe the AbdomenCT-1K dataset will promote future in-depth research towards clinical applicable abdominal organ segmentation methods. The datasets, codes, and trained models are publicly available at https://github.com/JunMa11/AbdomenCT-1K.

The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification

Abstract: The BraTS 2021 challenge celebrates its 10th anniversary and is jointly organized by the Radiological Society of North America (RSNA), the American Society of Neuroradiology (ASNR), and the Medical Image Computing and Computer Assisted Interventions (MICCAI) society. Since its inception, BraTS has been focusing on being a common benchmarking venue for brain glioma segmentation algorithms, with well-curated multi-institutional multi-parametric magnetic resonance imaging (mpMRI) data. Gliomas are the most common primary malignancies of the central nervous system, with varying degrees of aggressiveness and prognosis. The RSNA-ASNR-MICCAI BraTS 2021 challenge targets the evaluation of computational algorithms assessing the same tumor compartmentalization, as well as the underlying tumor's molecular characterization, in pre-operative baseline mpMRI data from 2,040 patients. Specifically, the two tasks that BraTS 2021 focuses on are: a) the segmentation of the histologically distinct brain tumor sub-regions, and b) the classification of the tumor's O[6]-methylguanine-DNA methyltransferase (MGMT) promoter methylation status. The performance evaluation of all participating algorithms in BraTS 2021 will be conducted through the Sage Bionetworks Synapse platform (Task 1) and Kaggle (Task 2), concluding in distributing to the top ranked participants monetary awards of $60,000 collectively.

BUSIS: A Benchmark for Breast Ultrasound Image Segmentation

Abstract: Breast ultrasound (BUS) image segmentation is challenging and critical for BUS computer-aided diagnosis (CAD) systems. Many BUS segmentation approaches have been studied in the last two decades, but the performances of most approaches have been assessed using relatively small private datasets with different quantitative metrics, which results in a discrepancy in performance comparison. Therefore, there is a pressing need for building a benchmark to compare existing methods using a public dataset objectively, to determine the performance of the best breast tumor segmentation algorithm available today, and to investigate what segmentation strategies are valuable in clinical practice and theoretical study. In this work, a benchmark for B-mode breast ultrasound image segmentation is presented. In the benchmark, (1) we collected 562 breast ultrasound images and proposed standardized procedures to obtain accurate annotations using four radiologists; (2) we extensively compared the performance of 16 state-of-the-art segmentation methods and demonstrated that most deep learning-based approaches achieved high dice similarity coefficient values (DSC ≥ 0.90) and outperformed conventional approaches; (3) we proposed the losses-based approach to evaluate the sensitivity of semi-automatic segmentation to user interactions; and (4) the successful segmentation strategies and possible future improvements were discussed in details.

CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation

Abstract: Segmentation of abdominal organs has been a comprehensive, yet unresolved, research field for many years. In the last decade, intensive developments in deep learning (DL) have introduced new state-of-the-art segmentation systems. In order to expand the knowledge on these topics, the CHAOS - Combined (CT-MR) Healthy Abdominal Organ Segmentation challenge has been organized in conjunction with IEEE International Symposium on Biomedical Imaging (ISBI), 2019, in Venice, Italy. CHAOS provides both abdominal CT and MR data from healthy subjects for single and multiple abdominal organ segmentation. Five different but complementary tasks have been designed to analyze the capabilities of current approaches from multiple perspectives. The results are investigated thoroughly, compared with manual annotations and interactive methods. The analysis shows that the performance of DL models for single modality (CT / MR) can show reliable volumetric analysis performance (DICE: 0.98 $\pm$ 0.00 / 0.95 $\pm$ 0.01) but the best MSSD performance remain limited (21.89 $\pm$ 13.94 / 20.85 $\pm$ 10.63 mm). The performances of participating models decrease significantly for cross-modality tasks for the liver (DICE: 0.88 $\pm$ 0.15 MSSD: 36.33 $\pm$ 21.97 mm) and all organs (DICE: 0.85 $\pm$ 0.21 MSSD: 33.17 $\pm$ 38.93 mm). Despite contrary examples on different applications, multi-tasking DL models designed to segment all organs seem to perform worse compared to organ-specific ones (performance drop around 5\%). Besides, such directions of further research for cross-modality segmentation would significantly support real-world clinical applications. Moreover, having more than 1500 participants, another important contribution of the paper is the analysis on shortcomings of challenge organizations such as the effects of multiple submissions and peeking phenomena.

An automatic multi-tissue human fetal brain segmentation benchmark using the Fetal Tissue Annotation Dataset

Abstract: It is critical to quantitatively analyse the developing human fetal brain in order to fully understand neurodevelopment in both normal fetuses and those with congenital disorders. To facilitate this analysis, automatic multi-tissue fetal brain segmentation algorithms are needed, which in turn requires open databases of segmented fetal brains. Here we introduce a publicly available database of 50 manually segmented pathological and non-pathological fetal magnetic resonance brain volume reconstructions across a range of gestational ages (20 to 33 weeks) into 7 different tissue categories (external cerebrospinal fluid, grey matter, white matter, ventricles, cerebellum, deep grey matter, brainstem/spinal cord). In addition, we quantitatively evaluate the accuracy of several automatic multi-tissue segmentation algorithms of the developing human fetal brain. Four research groups participated, submitting a total of 10 algorithms, demonstrating the benefits the database for the development of automatic algorithms.

Deep Placental Vessel Segmentation for Fetoscopic Mosaicking

Abstract: During fetoscopic laser photocoagulation, a treatment for twin-to-twin transfusion syndrome (TTTS), the clinician first identifies abnormal placental vascular connections and laser ablates them to regulate blood flow in both fetuses. The procedure is challenging due to the mobility of the environment, poor visibility in amniotic fluid, occasional bleeding, and limitations in the fetoscopic field-of-view and image quality. Ideally, anastomotic placental vessels would be automatically identified, segmented and registered to create expanded vessel maps to guide laser ablation, however, such methods have yet to be clinically adopted. We propose a solution utilising the U-Net architecture for performing placental vessel segmentation in fetoscopic videos. The obtained vessel probability maps provide sufficient cues for mosaicking alignment by registering consecutive vessel maps using the direct intensity-based technique. Experiments on 6 different in vivo fetoscopic videos demonstrate that the vessel intensity-based registration outperformed image intensity-based registration approaches showing better robustness in qualitative and quantitative comparison. We additionally reduce drift accumulation to negligible even for sequences with up to 400 frames and we incorporate a scheme for quantifying drift error in the absence of the ground-truth. Our paper provides a benchmark for fetoscopy placental vessel segmentation and registration by contributing the first in vivo vessel segmentation and fetoscopic videos dataset.

Early Detection of Myocardial Infarction in Low-Quality Echocardiography

Abstract:
Myocardial infarction (MI), or commonly known as <italic>heart attack</italic>, is a life-threatening health problem worldwide from which 32.4 million people suffer each year. Early diagnosis and treatment of MI are crucial to prevent further heart tissue damages or death. The earliest and most reliable sign of ischemia is <italic>regional wall motion abnormality</italic> (RWMA) of the affected part of the ventricular muscle. Echocardiography can easily, inexpensively, and non-invasively exhibit the RWMA. In this article, we introduce a three-phase approach for early MI detection in low-quality echocardiography: 1) segmentation of the <italic>entire</italic> left ventricle (LV) wall using a state-of-the-art deep learning model, 2) analysis of the segmented LV wall by feature engineering, and 3) early MI detection. The main contributions of this study are highly accurate segmentation of the LV wall from low-quality echocardiography, pseudo labeling approach for ground-truth formation of the unannotated LV wall, and the first public echocardiographic dataset (HMC-QU)<xref ref-type="fn" rid="fn1"><italic>a</italic></xref> MI detection. Furthermore, the outputs of the proposed approach can significantly help cardiologists for a better assessment of the LV wall characteristics. The proposed approach has achieved 95.72% sensitivity and 99.58% specificity for the LV wall segmentation, and 85.97% sensitivity, 74.03% specificity, and 86.85% precision for MI detection on the HMC-QU dataset.<fn id="fn1">a
The benchmark HMC-QU dataset is publicly shared at the repository <uri>https://www.kaggle.com/aysendegerli/hmcqu-dataset.for</uri>
</fn>

ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset

Abstract: Magnetic resonance imaging (MRI) is a central modality for stroke imaging. It is used upon patient admission to make treatment decisions such as selecting patients for intravenous thrombolysis or endovascular therapy. MRI is later used in the duration of hospital stay to predict outcome by visualizing infarct core size and location. Furthermore, it may be used to characterize stroke etiology, e.g. differentiation between (cardio)-embolic and non-embolic stroke. Computer based automated medical image processing is increasingly finding its way into clinical routine. Previous iterations of the Ischemic Stroke Lesion Segmentation (ISLES) challenge have aided in the generation of identifying benchmark methods for acute and sub-acute ischemic stroke lesion segmentation. Here we introduce an expert-annotated, multicenter MRI dataset for segmentation of acute to subacute stroke lesions. This dataset comprises 400 multi-vendor MRI cases with high variability in stroke lesion size, quantity and location. It is split into a training dataset of n=250 and a test dataset of n=150. All training data will be made publicly available. The test dataset will be used for model validation only and will not be released to the public. This dataset serves as the foundation of the ISLES 2022 challenge with the goal of finding algorithmic methods to enable the development and benchmarking of robust and accurate segmentation algorithms for ischemic stroke.

The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge

Abstract: There is a large body of literature linking anatomic and geometric characteristics of kidney tumors to perioperative and oncologic outcomes. Semantic segmentation of these tumors and their host kidneys is a promising tool for quantitatively characterizing these lesions, but its adoption is limited due to the manual effort required to produce high-quality 3D segmentations of these structures. Recently, methods based on deep learning have shown excellent results in automatic 3D segmentation, but they require large datasets for training, and there remains little consensus on which methods perform best. The 2019 Kidney and Kidney Tumor Segmentation challenge (KiTS19) was a competition held in conjunction with the 2019 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) which sought to address these issues and stimulate progress on this automatic segmentation problem. A training set of 210 cross sectional CT images with kidney tumors was publicly released with corresponding semantic segmentation masks. 106 teams from five continents used this data to develop automated systems to predict the true segmentation masks on a test set of 90 CT images for which the corresponding ground truth segmentations were kept private. These predictions were scored and ranked according to their average So rensen-Dice coefficient between the kidney and tumor across all 90 cases. The winning team achieved a Dice of 0.974 for kidney and 0.851 for tumor, approaching the inter-annotator performance on kidney (0.983) but falling short on tumor (0.923). This challenge has now entered an "open leaderboard" phase where it serves as a challenging benchmark in 3D semantic segmentation.

Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm

Abstract: Recent analysis identified distinct genomic subtypes of lower-grade glioma tumors which are associated with shape features. In this study, we propose a fully automatic way to quantify tumor imaging characteristics using deep learning-based segmentation and test whether these characteristics are predictive of tumor genomic subtypes. We used preoperative imaging and genomic data of 110 patients from 5 institutions with lower-grade gliomas from The Cancer Genome Atlas. Based on automatic deep learning segmentations, we extracted three features which quantify two-dimensional and three-dimensional characteristics of the tumors. Genomic data for the analyzed cohort of patients consisted of previously identified genomic clusters based on IDH mutation and 1p/19q co-deletion, DNA methylation, gene expression, DNA copy number, and microRNA expression. To analyze the relationship between the imaging features and genomic clusters, we conducted the Fisher exact test for 10 hypotheses for each pair of imaging feature and genomic subtype. To account for multiple hypothesis testing, we applied a Bonferroni correction. P-values lower than 0.005 were considered statistically significant. We found the strongest association between RNASeq clusters and the bounding ellipsoid volume ratio ($p<0.0002$) and between RNASeq clusters and margin fluctuation ($p<0.005$). In addition, we identified associations between bounding ellipsoid volume ratio and all tested molecular subtypes ($p<0.02$) as well as between angular standard deviation and RNASeq cluster ($p<0.02$). In terms of automatic tumor segmentation that was used to generate the quantitative image characteristics, our deep learning algorithm achieved a mean Dice coefficient of 82% which is comparable to human performance.

Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge

Abstract: Automatic detection of pulmonary nodules in thoracic computed tomography (CT) scans has been an active area of research for the last two decades. However, there have only been few studies that provide a comparative performance evaluation of different systems on a common database. We have therefore set up the LUNA16 challenge, an objective evaluation framework for automatic nodule detection algorithms using the largest publicly available reference database of chest CT scans, the LIDC-IDRI data set. In LUNA16, participants develop their algorithm and upload their predictions on 888 CT scans in one of the two tracks: 1) the complete nodule detection track where a complete CAD system should be developed, or 2) the false positive reduction track where a provided set of nodule candidates should be classified. This paper describes the setup of LUNA16 and presents the results of the challenge so far. Moreover, the impact of combining individual systems on the detection performance was also investigated. It was observed that the leading solutions employed convolutional networks and used the provided set of nodule candidates. The combination of these solutions achieved an excellent sensitivity of over 95% at fewer than 1.0 false positives per scan. This highlights the potential of combining algorithms to improve the detection performance. Our observer study with four expert readers has shown that the best system detects nodules that were missed by expert readers who originally annotated the LIDC-IDRI data. We released this set of additional nodules for further development of CAD systems.

A large annotated medical image dataset for the development and evaluation of segmentation algorithms

Abstract: Semantic segmentation of medical images aims to associate a pixel with a label in a medical image without human initialization. The success of semantic segmentation algorithms is contingent on the availability of high-quality imaging data with corresponding labels provided by experts. We sought to create a large collection of annotated medical image datasets of various clinically relevant anatomies available under open source license to facilitate the development of semantic segmentation algorithms. Such a resource would allow: 1) objective assessment of general-purpose segmentation methods through comprehensive benchmarking and 2) open and free access to medical image data for any researcher interested in the problem domain. Through a multi-institutional effort, we generated a large, curated dataset representative of several highly variable segmentation tasks that was used in a crowd-sourced challenge - the Medical Segmentation Decathlon held during the 2018 Medical Image Computing and Computer Aided Interventions Conference in Granada, Spain. Here, we describe these ten labeled image datasets so that these data may be effectively reused by the research community.

ROSE: A Retinal OCT-Angiography Vessel Segmentation Dataset and New Model

Abstract: Optical Coherence Tomography Angiography (OCTA) is a non-invasive imaging technique that has been increasingly used to image the retinal vasculature at capillary level resolution. However, automated segmentation of retinal vessels in OCTA has been under-studied due to various challenges such as low capillary visibility and high vessel complexity, despite its significance in understanding many vision-related diseases. In addition, there is no publicly available OCTA dataset with manually graded vessels for training and validation of segmentation algorithms. To address these issues, for the first time in the field of retinal image analysis we construct a dedicated Retinal OCTA SEgmentation dataset (ROSE), which consists of 229 OCTA images with vessel annotations at either centerline-level or pixel level. This dataset with the source code has been released for public access to assist researchers in the community in undertaking research in related topics. Secondly, we introduce a novel split-based coarse-to-fine vessel segmentation network for OCTA images (OCTA-Net), with the ability to detect thick and thin vessels separately. In the OCTA-Net, a split-based coarse segmentation module is first utilized to produce a preliminary confidence map of vessels, and a split-based refined segmentation module is then used to optimize the shape/contour of the retinal microvasculature. We perform a thorough evaluation of the state-of-the-art vessel segmentation models and our OCTA-Net on the constructed ROSE dataset. The experimental results demonstrate that our OCTA-Net yields better vessel segmentation performance in OCTA than both traditional and other deep learning methods. In addition, we provide a fractal dimension analysis on the segmented microvasculature, and the statistical analysis demonstrates significant differences between the healthy control and Alzheimer’s Disease group. This consolidates that the analysis of retinal microvasculature may offer a new scheme to study various neurodegenerative diseases.

We assembled the following set of datasets to train UniverSeg. For the relative size of datasets, we have included the number of unique scans (subject and modality pairs) that each dataset has.

Datasets.
MegaMedical features a wide array of biomedical domains, such as eyes {{cite:3757e16}}, {{cite:a2a3af3}}, {{cite:8a69d2b}}, {{cite:15e6ac5}}, {{cite:dd8fa75}}, lungs {{cite:8a76233}}, {{cite:f63c296}}, {{cite:16ab328}}, spine vertebrae {{cite:b5bfb8e}}, white blood cells {{cite:c3b2f6b}}, abdominal {{cite:180fdf5}}, {{cite:53ecf89}}, {{cite:3775317}}, {{cite:ed09a34}}, {{cite:2832bb1}}, {{cite:780a859}}, {{cite:18ca277}}, {{cite:c7270b1}}, {{cite:f586d57}}, {{cite:ca27ac5}}, {{cite:9522ebe}}, {{cite:1167446}}, {{cite:16ab328}}, and brain {{cite:eafe561}}, {{cite:43b1523}}, {{cite:a7fd3b3}}, {{cite:619acf0}}, {{cite:e6df96d}}, {{cite:47c14b1}}, {{cite:53cb564}}, {{cite:109fdea}}, {{cite:16ab328}}, among others.
Supplemental Table {{table:881f95d0-4658-4226-ab22-f0e8f69a9996}}  provides a detailed list of MegaMedical datasets.
Acquisition details, subject age ranges, and health conditions are different for each dataset.
We provide preprocessing and data normalization details in supplemental Section .


Electronic pathology textbooksPEIR Digital Library

Medical Segmentation Decathlon{{cite:ac0f04b}},NIH Chest X-ray{{cite:004b57b}},CHAOS{{cite:048f533}}:- Chest X-rays/CTs- Abdomen CTs/MRIs- Head CTs/MRIs- Neck CTs- Pelvic cavity CTs

- Color- Location- Appearance- Shape- etc.

- Organ- Position- Knowledge Graph- Abnormality- Modality- Plane- Quality- Color- Size- Shape

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

Abstract: Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain.

PathVQA: 30000+ Questions for Medical Visual Question Answering

Abstract: Is it possible to develop an "AI Pathologist" to pass the board-certified examination of the American Board of Pathology? To achieve this goal, the first step is to create a visual question answering (VQA) dataset where the AI agent is presented with a pathology image together with a question and is asked to give the correct answer. Our work makes the first attempt to build such a dataset. Different from creating general-domain VQA datasets where the images are widely accessible and there are many crowdsourcing workers available and capable of generating question-answer pairs, developing a medical VQA dataset is much more challenging. First, due to privacy concerns, pathology images are usually not publicly available. Second, only well-trained pathologists can understand pathology images, but they barely have time to help create datasets for AI research. To address these challenges, we resort to pathology textbooks and online digital libraries. We develop a semi-automated pipeline to extract pathology images and captions from textbooks and generate question-answer pairs from captions using natural language processing. We collect 32,799 open-ended questions from 4,998 pathology images where each question is manually checked to ensure correctness. To our best knowledge, this is the first dataset for pathology VQA. Our dataset will be released publicly to promote research in medical VQA.

Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering

Abstract: Medical visual question answering (Med-VQA) has tremendous potential in healthcare. However, the development of this technology is hindered by the lacking of publicly-available and high-quality labeled datasets for training and evaluation. In this paper, we present a large bilingual dataset, SLAKE, with comprehensive semantic labels annotated by experienced physicians and a new structural medical knowledge base for Med-VQA. Besides, SLAKE includes richer modalities and covers more human body parts than the currently available dataset. We show that SLAKE can be used to facilitate the development and evaluation of Med-VQA systems. The dataset can be downloaded from http://www.med-vqa.com/slake.

Overview of the medical VQA datasets and their main characteristics. Visual Genome, VQA 2.0, and OK-VQA are general-domain VQA datasets listed here for comparison. The medical VQA datasets are presented in chronological order.

To the best of our knowledge, there are 8 public-available medical VQA datasets up to date: VQA-MED-2018 {{cite:72d2041}}, VQA-RAD {{cite:e5f1d05}}, VQA-MED-2019 {{cite:9445067}}, RadVisDial {{cite:454c921}}, PathVQA {{cite:cad8f1b}}, VQA-MED-2020 {{cite:88918cc}}, SLAKE {{cite:29f7abc}}, and VQA-MED-2021 {{cite:0470aa4}} (in chronological order). The dataset's details are summarized in Table {{table:5a000529-a230-4126-8210-1b3c99df3e0c}} . In the following paragraphs, we provide an overview of the QA pairs collection.

In the above sections, we present 8 medical VQA datasets about their quantity, data source, QA creation, and question categories. As shown in Table {{table:5a000529-a230-4126-8210-1b3c99df3e0c}} , we also list three general VQA datasets for comparison. The image amounts of medical VQA datasets range from 315 to 91,060, while the number of QA pairs ranges from 1 QA pair per image to 10 QA pairs per image. The imaging modality includes chest X-ray, CT, MRI, and pathology. Except for RadVisDial, the medical VQA datasets are significantly smaller than the general VQA datasets in quantity. For the QA pair creation, the medical VQA uses synthetic creation more frequently than the general domain VQA. The question categories of medical VQA and general VQA are quite different. Besides the common categories of object and attribute, the general VQA research extends their problem to objects' relationship and external knowledge, while the medical VQA research tends to image findings.

Besides dataset design, question diversity raises a challenge in method development. To answer the diverse questions, the medical VQA systems require various reasoning abilities. For example, as a sample shown in Table {{table:bb00c253-d23b-4406-819b-d0f93559ea2c}} , to answer question “What is the function of the rightmost organ in this picture?”, the model should understand the region described, identify the organ in the region of interest, and finally answer the function of the organ. Besides the basic image and language understanding, medical domain knowledge is a critical ability required, which includes knowledge of anatomical structures, medical procedures, diseases, medical imaging modalities, treatment options, and clinical practices. More specially, for the question categories in Table {{table:5a000529-a230-4126-8210-1b3c99df3e0c}} , Modality, Plane need knowledge about radiology examination; Organ, System, Abnormality, Object/Condition Presence need knowledge about human anatomy and medicine; Positional Reasoning, Color, Size, Attribute Other, Counting need general knowledge and reasoning; Knowledge Graph need to combine the upon knowledge with the knowledge triplets given in dataset. Therefore, to address question categories correspondingly, the future medical VQA should be equipped with computer-aid diagnosis, general language understanding, reasoning, knowledge integration, and contextual understanding.


Automatic Retinal Blood Vessel Segmentation Based on Fully Convolutional Neural Networks

Abstract: Automated retinal vessel segmentation technology has become an important tool for disease screening and diagnosis in clinical medicine. However, most of the available methods of retinal vessel segmentation still have problems such as poor accuracy and low generalization ability. This is because the symmetrical and asymmetrical patterns between blood vessels are complicated, and the contrast between the vessel and the background is relatively low due to illumination and pathology. Robust vessel segmentation of the retinal image is essential for improving the diagnosis of diseases such as vein occlusions and diabetic retinopathy. Automated retinal vein segmentation remains a challenging task. In this paper, we proposed an automatic retinal vessel segmentation framework using deep fully convolutional neural networks (FCN), which integrate novel methods of data preprocessing, data augmentation, and full convolutional neural networks. It is an end-to-end framework that automatically and efficiently performs retinal vessel segmentation. The framework was evaluated on three publicly available standard datasets, achieving F1 score of 0.8321, 0.8531, and 0.8243, an average accuracy of 0.9706, 0.9777, and 0.9773, and average area under the Receiver Operating Characteristic (ROC) curve of 0.9880, 0.9923 and 0.9917 on the DRIVE, STARE, and CHASE_DB1 datasets, respectively. The experimental results show that our proposed framework achieves state-of-the-art vessel segmentation performance in all three benchmark tests.

A fully convolutional neural network based structured prediction approach towards the retinal vessel segmentation

Abstract: Automatic segmentation of retinal blood vessels from fundus images plays an important role in the computer aided diagnosis of retinal diseases. The task of blood vessel segmentation is challenging due to the extreme variations in morphology of the vessels against noisy background. In this paper, we formulate the segmentation task as a multi-label inference task and utilize the implicit advantages of the combination of convolutional neural networks and structured prediction. Our proposed convolutional neural network based model achieves strong performance and significantly outperforms the state-of-the-art for automatic retinal blood vessel segmentation on DRIVE dataset with 95.33% accuracy and 0.974 AUC score.

A supervised blood vessel segmentation technique for digital Fundus images using Zernike Moment based features

Abstract: This paper proposes a new supervised method for blood vessel segmentation using Zernike moment-based shape descriptors. The method implements a pixel wise classification by computing a 11-D feature vector comprising of both statistical (gray-level) features and shape-based (Zernike moment) features. Also the feature set contains optimal coefficients of the Zernike Moments which were derived based on the maximum differentiability between the blood vessel and background pixels. A manually selected training points obtained from the training set of the DRIVE dataset, covering all possible manifestations were used for training the ANN-based binary classifier. The method was evaluated on unknown test samples of DRIVE and STARE databases and returned accuracies of 0.945 and 0.9486 respectively, outperforming other existing supervised learning methods. Further, the segmented outputs were able to cover thinner blood vessels better than previous methods, aiding in early detection of pathologies.

Multilevel and Multiscale Deep Neural Network for Retinal Blood Vessel Segmentation

Abstract: Retinal blood vessel segmentation influences a lot of blood vessel-related disorders such as diabetic retinopathy, hypertension, cardiovascular and cerebrovascular disorders, etc. It is found that vessel segmentation using a convolutional neural network (CNN) showed increased accuracy in feature extraction and vessel segmentation compared to the classical segmentation algorithms. CNN does not need any artificial handcrafted features to train the network. In the proposed deep neural network (DNN), a better pre-processing technique and multilevel/multiscale deep supervision (DS) layers are being incorporated for proper segmentation of retinal blood vessels. From the first four layers of the VGG-16 model, multilevel/multiscale deep supervision layers are formed by convolving vessel-specific Gaussian convolutions with two different scale initializations. These layers output the activation maps that are capable to learn vessel-specific features at multiple scales, levels, and depth. Furthermore, the receptive field of these maps is increased to obtain the symmetric feature maps that provide the refined blood vessel probability map. This map is completely free from the optic disc, boundaries, and non-vessel background. The segmented results are tested on Digital Retinal Images for Vessel Extraction (DRIVE), STructured Analysis of the Retina (STARE), High-Resolution Fundus (HRF), and real-world retinal datasets to evaluate its performance. This proposed model achieves better sensitivity values of 0.8282, 0.8979 and 0.8655 in DRIVE, STARE and HRF datasets with acceptable specificity and accuracy performance metrics.

A Hierarchical Image Matting Model for Blood Vessel Segmentation in Fundus Images

Abstract: In this paper, a hierarchical image matting model is proposed to extract blood vessels from fundus images. More specifically, a hierarchical strategy is integrated into the image matting model for blood vessel segmentation. Normally, the matting models require a user specified <italic>trimap</italic>, which separates the input image into three regions: the foreground, background, and unknown regions. However, creating a user specified trimap is laborious for vessel segmentation tasks. In this paper, we propose a method that first generates trimap automatically by utilizing region features of blood vessels, then applies a hierarchical image matting model to extract the vessel pixels from the unknown regions. The proposed method has low calculation time and outperforms many other state-of-art supervised and unsupervised methods. It achieves a vessel segmentation accuracy of 96.0%, 95.7%, and 95.1% in an average time of <inline-formula> <tex-math notation="LaTeX">$10.72s$ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$15.74s$ </tex-math></inline-formula>, and <inline-formula> <tex-math notation="LaTeX">$50.71s$ </tex-math></inline-formula> on images from three publicly available fundus image datasets DRIVE, STARE, and CHASE_DB1, respectively.

A New Approach to Segment Both Main and Peripheral Retinal Vessels Based on Gray-Voting and Gaussian Mixture Model

Abstract: Vessel segmentation in retinal fundus images is a preliminary step to clinical diagnosis for some systemic diseases and some eye diseases. The performances of existing methods for segmenting small vessels which are usually of more importance than the main vessels in a clinical diagnosis are not satisfactory in clinical use. In this paper, we present a method for both main and peripheral vessel segmentation. A local gray-level change enhancement algorithm called gray-voting is used to enhance the small vessels, while a two-dimensional Gabor wavelet is used to extract the main vessels. We fuse the gray-voting results with the 2D-Gabor filter results as pre-processing outcome. A Gaussian mixture model is then used to extract vessel clusters from the pre-processing outcome, while small vessels fragments are obtained using another gray-voting process, which complements the vessel cluster extraction already performed. At the last step, we eliminate the fragments that do not belong to the vessels based on the shape of the fragments. We evaluated the approach with two publicly available DRIVE (Staal et al., 2004) and STARE (Hoover et at., 2000) datasets with manually segmented results. For the STARE dataset, when using the second manually segmented results which include much more small vessels than the first manually segmented results as the “gold standard,” this approach achieved an average sensitivity, accuracy and specificity of 65.0%, 92.1% and 97.0%, respectively. The sensitivities of this approach were much higher than those of the other existing methods, with comparable specificities; these results thus demonstrated that this approach was sensitive to detection of small vessels.

Robust Vessel Segmentation in Fundus Images

Abstract: One of the most common modalities to examine the human eye is the eye-fundus photograph. The evaluation of fundus photographs is carried out by medical experts during time-consuming visual inspection. Our aim is to accelerate this process using computer aided diagnosis. As a first step, it is necessary to segment structures in the images for tissue differentiation. As the eye is the only organ, where the vasculature can be imaged in an in vivo and noninterventional way without using expensive scanners, the vessel tree is one of the most interesting and important structures to analyze. The quality and resolution of fundus images are rapidly increasing. Thus, segmentation methods need to be adapted to the new challenges of high resolutions. In this paper, we present a method to reduce calculation time, achieve high accuracy, and increase sensitivity compared to the original Frangi method. This method contains approaches to avoid potential problems like specular reflexes of thick vessels. The proposed method is evaluated using the STARE and DRIVE databases and we propose a new high resolution fundus database to compare it to the state-of-the-art algorithms. The results show an average accuracy above 94% and low computational needs. This outperforms state-of-the-art methods.

Adaptive Thresholding Technique for Retinal Vessel Segmentation Based on GLCM-Energy Information

Abstract: Although retinal vessel segmentation has been extensively researched, a robust and time efficient segmentation method is highly needed. This paper presents a local adaptive thresholding technique based on gray level cooccurrence matrix- (GLCM-) energy information for retinal vessel segmentation. Different thresholds were computed using GLCM-energy information. An experimental evaluation on DRIVE database using the grayscale intensity and Green Channel of the retinal image demonstrates the high performance of the proposed local adaptive thresholding technique. The maximum average accuracy rates of 0.9511 and 0.9510 with maximum average sensitivity rates of 0.7650 and 0.7641 were achieved on DRIVE and STARE databases, respectively. When compared to the widely previously used techniques on the databases, the proposed adaptive thresholding technique is time efficient with a higher average sensitivity and average accuracy rates in the same range of very good specificity.

The improved ranking based on adjusted performance scores, without outliers.

The adjustment was applied to the reported scores of all methods we identified to be evaluated using all pixels. The worst case of the adjustment is {{formula:f8ac3ce0-b5ce-4dcf-a6b8-243710a56cb5}}  with the average of {{formula:f9dc278f-b80d-4e66-b31e-499d01d51e6a}} . As the errors are an order of magnitude smaller than the bias, we believe that the ranking based on the adjusted scores in Table {{table:5aa236dc-2c1f-46a1-8370-9651c623390c}}  is a more reliable baseline for the field than the original ranking in Table {{table:8470c7c5-725e-4464-bfea-392f48722812}} .

In this study, we investigated the coherence of reported performance scores in the segmentation of vessels in retinal images. We collected 100 influential papers reporting accuracy, sensitivity, and specificity for the publicly available data set DRIVE. Based on textual references to two corner cases of evaluation methodologies (evaluation under the FoV mask or using all pixels of the images), we developed numerical methods to test the consistency of the reported scores with the two assumed corner cases (subsections REF , REF ). Based on results of consistency tests, we attempted to remove bias and make the scores comparable to create a new baseline ranking (Section REF ). Section  summarizes various findings and insights and also discusses the limitations of our analysis. The insights clearly show that there is no consensus in the field on the evaluation methodology. However, in most of the papers, including reviews of the field, the rankings and the objective evaluation of operating principles are biassed by the direct comparison of non-comparable figures. Based on the papers we selected and the adjustment we implemented, the current state of the art accuracy is 0.9582 (evaluated in the FoV region), which is about 1.1% higher than the performance of annotator #2. Despite the limitations of our method, we believe that the adjusted ranking in Table {{table:5aa236dc-2c1f-46a1-8370-9651c623390c}}  provides a more realistic view of the state of the art than any ranking based on the direct comparison of reported figures.


Case numbers, demographics, health conditions, social activities, environmental factors, etc.

A SIDARTHE Model of COVID-19 Epidemic in Italy

Abstract: In late December 2019, a novel strand of Coronavirus (SARS-CoV-2) causing a severe, potentially fatal respiratory syndrome (COVID-19) was identified in Wuhan, Hubei Province, China and is causing outbreaks in multiple world countries, soon becoming a pandemic. Italy has now become the most hit country outside of Asia: on March 16, 2020, the Italian Civil Protection documented a total of 27980 confirmed cases and 2158 deaths of people tested positive for SARS-CoV-2. In the context of an emerging infectious disease outbreak, it is of paramount importance to predict the trend of the epidemic in order to plan an effective control strategy and to determine its impact. This paper proposes a new epidemic model that discriminates between infected individuals depending on whether they have been diagnosed and on the severity of their symptoms. The distinction between diagnosed and non-diagnosed is important because non-diagnosed individuals are more likely to spread the infection than diagnosed ones, since the latter are typically isolated, and can explain misperceptions of the case fatality rate and of the seriousness of the epidemic phenomenon. Being able to predict the amount of patients that will develop life-threatening symptoms is important since the disease frequently requires hospitalisation (and even Intensive Care Unit admission) and challenges the healthcare system capacity. We show how the basic reproduction number can be redefined in the new framework, thus capturing the potential for epidemic containment. Simulation results are compared with real data on the COVID-19 epidemic in Italy, to show the validity of the model and compare different possible predicted scenarios depending on the adopted countermeasures.

Modeling strict age-targeted mitigation strategies for COVID-19

Abstract: We use a simple SIR-like epidemic model integrating known age-contact patterns for the United States to model the effect of age-targeted mitigation strategies for a COVID-19-like epidemic. We find that, among strategies which end with population immunity, strict age-targeted mitigation strategies have the potential to greatly reduce mortalities and ICU utilization for natural parameter choices.

Examples of COVID-19 Epidemiological Modeling.

Modeling COVID-19 epidemic transmission process.
Studies on modeling COVID-19 epidemic transmission mainly focus on evaluating the epidemiological attributes (e.g., infection rate, recovery rate, mortality, reproduction number, etc.), predicting the infection and death counts, and revealing the transmission, spread and outbreak trends under experimental or real-world scenarios. As illustrated in Table {{table:17c7346c-7e8b-4da6-b425-6087733bec9a}} , various compartmental models are available to characterize COVID-19. For example, the SIDARTHE compartmental model considers eight stages of infection: susceptible ({{formula:62c84ccb-aad5-44e9-aa30-f5471f2794f7}} ), infected ({{formula:30c24353-f894-48fd-81c1-45789cb39c49}} ), diagnosed ({{formula:562a84bf-6d2e-4924-bad8-935b42cec106}} ), ailing ({{formula:8534f279-29f1-438c-a451-1729686d5291}} ), recognized ({{formula:54bff66f-b839-4944-a75f-e4b533147504}} ), threatened ({{formula:b34229b1-35e5-4611-ac90-90469fcfb018}} ), healed ({{formula:298b6acb-e38b-47be-9edc-7d94b0fca96b}} ) and extinct ({{formula:304e617d-16d9-4b80-8640-b48fa7f07031}} ) to predict the course of the epidemic and to plan an effective control strategy {{cite:633edc4}}. A new compartment is introduced to the classic SIR model to quantify those who are symptomatic, quarantined infecteds {{cite:4fa3ac9}}. Further, a stochastic SHARUCD model framework contains seven compartments: susceptible ({{formula:6c7d36d9-63b8-492a-8d89-aa60582dbdca}} ), severe cases prone to hospitalization ({{formula:4085768a-c517-4068-85cf-529c0dd40674}} ), mild, sub-clinical or asymptomatic ({{formula:4dedad2e-89d4-44eb-8a60-26e06ddc970e}} ), recovered ({{formula:41625c28-4741-4f53-af5f-5ef4e222bcf0}} ), patients admitted to the intensive care units ({{formula:8c830edd-a6cd-49a2-8901-89bdc26f2166}} ), and the recorded cumulative positive cases ({{formula:e4473dea-4a22-4df6-9345-7e3eeed7bf0d}} ), which include all new positive cases for each class of {{formula:9ec4387d-b45b-4d6c-a07d-24a666823a52}} , {{formula:cf390e3c-6060-4630-877b-d2165c7fb773}} , {{formula:49a7e0a8-54be-4d73-8c07-0e02e32dd66a}} , {{formula:c4fd92f3-f884-4567-b7ce-fcfba876a415}} , and deceased ({{formula:fa437d0f-3393-4531-ae85-fe0d6a8d529e}} ) {{cite:7d1cb3a}}. In addition, several models involve new compartments to represent asymptomatic features to mild symptoms {{cite:7d1cb3a}}, {{cite:89687cc}} and undocumented cases {{cite:e45d83d}}.


Intelligent Intrusion Detection of Grey Hole and Rushing Attacks in Self-Driving Vehicular Networks

Abstract: Vehicular ad hoc networks (VANETs) play a vital role in the success of self-driving and semi self-driving vehicles, where they improve safety and comfort. Such vehicles depend heavily on external communication with the surrounding environment via data control and Cooperative Awareness Messages (CAMs) exchanges. VANETs are potentially exposed to a number of attacks, such as grey hole, black hole, wormhole and rushing attacks. This work presents an intelligent Intrusion Detection System (IDS) that relies on anomaly detection to protect the external communication system from grey hole and rushing attacks. These attacks aim to disrupt the transmission between vehicles and roadside units. The IDS uses features obtained from a trace file generated in a network simulator and consists of a feed-forward neural network and a support vector machine. Additionally, the paper studies the use of a novel systematic response, employed to protect the vehicle when it encounters malicious behaviour. Our simulations of the proposed detection system show that the proposed schemes possess outstanding detection rates with a reduction in false alarms. This safe mode response system has been evaluated using four performance metrics, namely, received packets, packet delivery ratio, dropped packets and the average end to end delay, under both normal and abnormal conditions.

Know Your Master: Driver Profiling-based Anti-theft Method

Abstract: Although many anti-theft technologies are implemented, auto-theft is still increasing. Also, security vulnerabilities of cars can be used for auto-theft by neutralizing anti-theft system. This keyless auto-theft attack will be increased as cars adopt computerized electronic devices more. To detect auto-theft efficiently, we propose the driver verification method that analyzes driving patterns using measurements from the sensor in the vehicle. In our model, we add mechanical features of automotive parts that are excluded in previous works, but can be differentiated by drivers' driving behaviors. We design the model that uses significant features through feature selection to reduce the time cost of feature processing and improve the detection performance. Further, we enrich the feature set by deriving statistical features such as mean, median, and standard deviation. This minimizes the effect of fluctuation of feature values per driver and finally generates the reliable model. We also analyze the effect of the size of sliding window on performance to detect the time point when the detection becomes reliable and to inform owners the theft event as soon as possible. We apply our model with real driving and show the contribution of our work to the literature of driver identification.

Investigations on Driver Unique Identification from Smartphone’s GPS Data Alone

Abstract: Driver identification is an emerging area of interest in vehicle telematics, automobile control, and insurance. Recent body of works indicates that it may be possible to uniquely identify a driver using multiple dedicated sensors. In this paper, we present an approach for driver identification using smartphone GPS data alone. For our experiments, we collected data from 38 drivers for two months. We quantified the driver’s natural style by extracting a set of 137 statistical features from data generated for each completed trip. The analysis shows that, for the “driver identification” problem, an average accuracy of 82.3% is achieved for driver groups of 4-5 drivers. This is comparable to the state of the arts where mostly a multisensor approach has been taken. Further, it is shown that certain behavioral attributes like high driving skill impact identification accuracy. We observe that Random Forest classifier offers the best results. These results have great implications for various stakeholders since the proposed method can identify a driver based on his/her naturalistic driving style which is quantified in terms of statistical parameters extracted from only GPS data.

Differentially Private Collaborative Intrusion Detection Systems For VANETs

Abstract: Vehicular ad hoc network (VANET) is an enabling technology in modern transportation systems for providing safety and valuable information, and yet vulnerable to a number of attacks from passive eavesdropping to active interfering. Intrusion detection systems (IDSs) are important devices that can mitigate the threats by detecting malicious behaviors. Furthermore, the collaborations among vehicles in VANETs can improve the detection accuracy by communicating their experiences between nodes. To this end, distributed machine learning is a suitable framework for the design of scalable and implementable collaborative detection algorithms over VANETs. One fundamental barrier to collaborative learning is the privacy concern as nodes exchange data among them. A malicious node can obtain sensitive information of other nodes by inferring from the observed data. In this paper, we propose a privacy-preserving machine-learning based collaborative IDS (PML-CIDS) for VANETs. The proposed algorithm employs the alternating direction method of multipliers (ADMM) to a class of empirical risk minimization (ERM) problems and trains a classifier to detect the intrusions in the VANETs. We use the differential privacy to capture the privacy notation of the PML-CIDS and propose a method of dual variable perturbation to provide dynamic differential privacy. We analyze theoretical performance and characterize the fundamental tradeoff between the security and privacy of the PML-CIDS. We also conduct numerical experiments using the NSL-KDD dataset to corroborate the results on the detection accuracy, security-privacy tradeoffs, and design.

Misbehavior-Aware On-Demand Collaborative Intrusion Detection System Using Distributed Ensemble Learning for VANET

Abstract: Vehicular ad hoc networks (VANETs) play an important role as enabling technology for future cooperative intelligent transportation systems (CITSs). Vehicles in VANETs share real-time information about their movement state, traffic situation, and road conditions. However, VANETs are susceptible to the cyberattacks that create life threatening situations and/or cause road congestion. Intrusion detection systems (IDSs) that rely on the cooperation between vehicles to detect intruders, were the most suggested security solutions for VANET. Unfortunately, existing cooperative IDSs (CIDSs) are vulnerable to the legitimate yet compromised collaborators that share misleading and manipulated information and disrupt the IDSs’ normal operation. As such, this paper proposes a misbehavior-aware on-demand collaborative intrusion detection system (MA-CIDS) based on the concept of distributed ensemble learning. That is, vehicles individually use the random forest algorithm to train local IDS classifiers and share their locally trained classifiers on-demand with the vehicles in their vicinity, which reduces the communication overhead. Once received, the performance of the classifiers is evaluated using the local testing dataset in the receiving vehicle. The evaluation values are used as a trustworthiness factor and used to rank the received classifiers. The classifiers that deviate much from the box-and-whisker plot lower boundary are excluded from the set of the collaborators. Then, each vehicle constructs an ensemble of weighted random forest-based classifiers that encompasses the locally and remotely trained classifiers. The outputs of the classifiers are aggregated using a robust weighted voting scheme. Extensive simulations were conducted utilizing the network security laboratory-knowledge discovery data mining (NSL-KDD) dataset to evaluate the performance of the proposed MA-CIDS model. The obtained results show that MA-CIDS performs better than the other existing models in terms of effectiveness and efficiency for VANET.

A Trust Model Using Edge Nodes and a Cuckoo Filter for Securing VANET under the NLoS Condition

Abstract: Trust, as a key element of security, has a vital role in securing vehicular ad-hoc networks (VANETs). Malicious and selfish nodes by generating inaccurate information, have undesirable impacts on the trustworthiness of the VANET environment. Obstacles also have a negative impact on data trustworthiness by restricting direct communication between nodes. In this study, a trust model based on plausibility, experience, and type of vehicle is presented to cope with inaccurate, incomplete and uncertainty data under both line of sight (LoS) and none-line of sight (NLoS) conditions. In addition, a model using the k-nearest neighbor (kNN) classification algorithm based on feature similarity and symmetry is developed to detect the NLoS condition. Radio signal strength indicator (RSSI), packet reception rate (PDR) and the distance between two vehicle nodes are the features used in the proposed kNN algorithm. Moreover, due to the big data generated in VANET, secure communication between vehicle and edge node is designed using the Cuckoo filter. All obtained results are validated through well-known evaluation measures such as precision, recall, overall accuracy, and communication overhead. The results indicate that the proposed trust model has a better performance as compared to the attack-resistant trust management (ART) scheme and weighted voting (WV) approach. Additionally, the proposed trust model outperforms both ART and WV approaches under different patterns of attack such as a simple attack, opinion tampering attack, and cunning attack. Monte-Carlo simulation results also prove validity of the proposed trust model.

Robust Deep Reinforcement Learning for Security and Safety in Autonomous Vehicle Systems

Abstract: To operate effectively in tomorrow's smart cities, autonomous vehicles (AVs) must rely on intra-vehicle sensors such as camera and radar as well as inter-vehicle communication. Such dependence on sensors and communication links exposes AVs to cyber-physical (CP) attacks by adversaries that seek to take control of the AVs by manipulating their data. Thus, to ensure safe and optimal AV dynamics control, the data processing functions at AVs must be robust to such CP attacks. To this end, in this paper, the state estimation process for monitoring AV dynamics, in presence of CP attacks, is analyzed and a novel adversarial deep reinforcement learning (RL) algorithm is proposed to maximize the robustness of AV dynamics control to CP attacks. The attacker's action and the AV's reaction to CP attacks are studied in a game-theoretic framework. In the formulated game, the attacker seeks to inject faulty data to AV sensor readings so as to manipulate the inter-vehicle optimal safe spacing and potentially increase the risk of AV accidents or reduce the vehicle flow on the roads. Meanwhile, the AV, acting as a defender, seeks to minimize the deviations of spacing so as to ensure robustness to the attacker's actions. Since the AV has no information about the attacker's action and due to the infinite possibilities for data value manipulations, the outcome of the players' past interactions are fed to long-short term memory (LSTM) blocks. Each player's LSTM block learns the expected spacing deviation resulting from its own action and feeds it to its RL algorithm. Then, the the attacker's RL algorithm chooses the action which maximizes the spacing deviation, while the AV's RL algorithm tries to find the optimal action that minimizes such deviation.

ML techniques applied to vehicular security (Part 01)

In Section IV, we have identified the use of ML-based solutions in five different security aspects including, driver fingerprinting, attack detection, misbehaviour or intrusion detection, trust computation, and privacy protection. Several ML architectures are explored in these areas where we observe supervised learning, reinforcement learning, and deep learning are popular learning algorithms in vehicular security applications, as shown in Table {{table:66d3670a-c2b9-4253-8e2f-7457c8e1c86b}}  and {{table:e84510ef-b078-4c1c-86c1-20faf3ccf9fc}} . In the context of driver fingerprinting, it is of utmost importance to preserve the privacy of driver data and prevent adversaries from tracking the vehicle by linking it to the driver's identification. The older techniques use the idea of hiding location and accessing resources/services without disclosing the true identity (i.e, using pseudonymity) of a driver to ensure privacy. With the advent of autonomous vehicles, true profiling of the driver is important. To enable this, ML techniques are used by researchers to authenticate the true identification and fingerprinting of a user/driver to make vehicles resilient to theft without hiding the true identity of drivers. We also note that supervised learning is the most-widely used ML technique for driver fingerprinting, as shown in Table {{table:66d3670a-c2b9-4253-8e2f-7457c8e1c86b}} . Some works in the literature, use unsupervised ML to cluster driver behaviour before performing classification and achieve better accuracies with the given models. It is observed that the data from smartphone sensors, vehicle sensors, and onboard electronic control units (i.e. using the OBD-II protocol) are the most commonly used data types for driver profiling using ML.


Graph Structural and Temporal Self-Attention

Custom Dataset: Sparse Scene, Dense Scene

Online Multiple Object Tracking with Cross-Task Synergy

Abstract: Modern online multiple object tracking (MOT) methods usually focus on two directions to improve tracking performance. One is to predict new positions in an incoming frame based on tracking information from previous frames, and the other is to enhance data association by generating more discriminative identity embeddings. Some works combined both directions within one framework but handled them as two individual tasks, thus gaining little mutual benefits. In this paper, we propose a novel unified model with synergy between position prediction and embedding association. The two tasks are linked by temporal-aware target attention and distractor attention, as well as identity-aware memory aggregation model. Specifically, the attention modules can make the prediction focus more on targets and less on distractors, therefore more reliable embeddings can be extracted accordingly for association. On the other hand, such reliable embeddings can boost identity-awareness through memory aggregation, hence strengthen attention modules and suppress drifts. In this way, the synergy between position prediction and embedding association is achieved, which leads to strong robustness to occlusions. Extensive experiments demonstrate the superiority of our proposed model over a wide range of existing methods on MOTChallenge benchmarks. Our code and models are publicly available at https://github.com/songguocode/TADAM.

Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation

Abstract: Multiple object tracking and segmentation requires detecting, tracking, and segmenting objects belonging to a set of given classes. Most approaches only exploit the temporal dimension to address the association problem, while relying on single frame predictions for the segmentation mask itself. We propose Prototypical Cross-Attention Network (PCAN), capable of leveraging rich spatio-temporal information for online multiple object tracking and segmentation. PCAN first distills a space-time memory into a set of prototypes and then employs cross-attention to retrieve rich information from the past frames. To segment each object, PCAN adopts a prototypical appearance module to learn a set of contrastive foreground and background prototypes, which are then propagated over time. Extensive experiments demonstrate that PCAN outperforms current video instance tracking and segmentation competition winners on both Youtube-VIS and BDD100K datasets, and shows efficacy to both one-stage and two-stage segmentation frameworks. Code and video resources are available at http://vis.xyz/pub/pcan.

DyGLIP: A Dynamic Graph Model with Link Prediction for Accurate Multi-Camera Multiple Object Tracking

Abstract: Multi-Camera Multiple Object Tracking (MC-MOT) is a significant computer vision problem due to its emerging applicability in several real-world applications. Despite a large number of existing works, solving the data association problem in any MC-MOT pipeline is arguably one of the most challenging tasks. Developing a robust MC-MOT system, however, is still highly challenging due to many practical issues such as inconsistent lighting conditions, varying object movement patterns, or the trajectory occlusions of the objects between the cameras. To address these problems, this work, therefore, proposes a new Dynamic Graph Model with Link Prediction (DyGLIP) approach 1 to solve the data association task. Compared to existing methods, our new model offers several advantages, including better feature representations and the ability to recover from lost tracks during camera transitions. Moreover, our model works gracefully regardless of the overlapping ratios between the cameras. Experimental results show that we out-perform existing MC-MOT algorithms by a large margin on several practical datasets. Notably, our model works favor-ably on online settings but can be extended to an incremental approach for large-scale datasets.

Summary of Attention related papers

To re-identify the occluded objects, attention is needed. Attention means we only consider the objects of interest by nullifying the background so that their features are remembered for long, even after occlusion. The summary of using attention module in MOT field is given in Table {{table:0eaf77c9-9cde-4b0a-a413-1dcd60848065}} .


Joint Future Frame/Label Propagation, Data Augmentation

Hierarchical Multi-Scale Attention for Semantic Segmentation

Abstract: Multi-scale inference is commonly used to improve the results of semantic segmentation. Multiple images scales are passed through a network and then the results are combined with averaging or max pooling. In this work, we present an attention-based approach to combining multi-scale predictions. We show that predictions at certain scales are better at resolving particular failures modes, and that the network learns to favor those scales for such cases in order to generate better predictions. Our attention mechanism is hierarchical, which enables it to be roughly 4x more memory efficient to train than other recent approaches. In addition to enabling faster training, this allows us to train with larger crop sizes which leads to greater model accuracy. We demonstrate the result of our method on two datasets: Cityscapes and Mapillary Vistas. For Cityscapes, which has a large number of weakly labelled images, we also leverage auto-labelling to improve generalization. Using our approach we achieve a new state-of-the-art results in both Mapillary (61.1 IOU val) and Cityscapes (85.1 IOU test).

Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation

Abstract: In this work, we introduce Panoptic-DeepLab, a simple, strong, and fast system for panoptic segmentation, aiming to establish a solid baseline for bottom-up methods that can achieve comparable performance of two-stage methods while yielding fast inference speed. In particular, Panoptic-DeepLab adopts the dual-ASPP and dual-decoder structures specific to semantic, and instance segmentation, respectively. The semantic segmentation branch is the same as the typical design of any semantic segmentation model (e.g., DeepLab), while the instance segmentation branch is class-agnostic, involving a simple instance center regression. As a result, our single Panoptic-DeepLab simultaneously ranks first at all three Cityscapes benchmarks, setting the new state-of-art of 84.2% mIoU, 39.0% AP, and 65.5% PQ on test set. Additionally, equipped with MobileNetV3, Panoptic-DeepLab runs nearly in real-time with a single 1025x2049 image (15.8 frames per second), while achieving a competitive performance on Cityscapes (54.1 PQ% on test set). On Mapillary Vistas test set, our ensemble of six models attains 42.7% PQ, outperforming the challenge winner in 2018 by a healthy margin of 1.5%. Finally, our Panoptic-DeepLab also performs on par with several top-down approaches on the challenging COCO dataset. For the first time, we demonstrate a bottom-up approach could deliver state-of-the-art results on panoptic segmentation.

Improving Semantic Segmentation via Video Propagation and Label Relaxation

Abstract: Semantic segmentation requires large amounts of pixel-wise annotations to learn accurate models. In this paper, we present a video prediction-based methodology to scale up training sets by synthesizing new training samples in order to improve the accuracy of semantic segmentation networks. We exploit video prediction models' ability to predict future frames in order to also predict future labels. A joint propagation strategy is also proposed to alleviate mis-alignments in synthesized samples. We demonstrate that training segmentation models on datasets augmented by the synthesized samples leads to significant improvements in accuracy. Furthermore, we introduce a novel boundary label relaxation technique that makes training robust to annotation noise and propagation artifacts along object boundaries. Our proposed methods achieve state-of-the-art mIoUs of 83.5% on Cityscapes and 82.9% on CamVid. Our single model, without model ensembles, achieves 72.8% mIoU on the KITTI semantic segmentation test set, which surpasses the winning entry of the ROB challenge 2018.

Hard Pixel Mining for Depth Privileged Semantic Segmentation

Abstract: Semantic segmentation has achieved remarkable progress but remains challenging due to the complex scene, object occlusion, and so on. Some research works have attempted to use extra information such as a depth map to help RGB based semantic segmentation because the depth map could provide complementary geometric cues. However, due to the inaccessibility of depth sensors, depth information is usually unavailable for the test images. In this paper, we leverage only the depth of training images as the privileged information to mine the hard pixels in semantic segmentation, in which depth information is only available for training images but not available for test images. Specifically, we propose a novel Loss Weight Module, which outputs a loss weight map by employing two depth-related measurements of hard pixels: Depth Prediction Error and Depth-aware Segmentation Error. The loss weight map is then applied to segmentation loss, with the goal of learning a more robust model by paying more attention to the hard pixels. Besides, we also explore a curriculum learning strategy based on the loss weight map. Meanwhile, to fully mine the hard pixels on different scales, we apply our loss weight module to multi-scale side outputs. Our hard pixels mining method achieves the state-of-the-art results on three benchmark datasets, and even outperforms the methods which need depth input during testing.

Global Aggregation then Local Distribution in Fully Convolutional Networks

Abstract: It has been widely proven that modelling long-range dependencies in fully convolutional networks (FCNs) via global aggregation modules is critical for complex scene understanding tasks such as semantic segmentation and object detection. However, global aggregation is often dominated by features of large patterns and tends to oversmooth regions that contain small patterns (e.g., boundaries and small objects). To resolve this problem, we propose to first use \emph{Global Aggregation} and then \emph{Local Distribution}, which is called GALD, where long-range dependencies are more confidently used inside large pattern regions and vice versa. The size of each pattern at each position is estimated in the network as a per-channel mask map. GALD is end-to-end trainable and can be easily plugged into existing FCNs with various global aggregation modules for a wide range of vision tasks, and consistently improves the performance of state-of-the-art object detection and instance segmentation approaches. In particular, GALD used in semantic segmentation achieves new state-of-the-art performance on Cityscapes test set with mIoU 83.3\%. Code is available at: \url{this https URL}

Cityscapes Pixel-Level Semantic Labeling Task Top Performing Models

In this section we will provide summary tables of the best performing models in semantic segmentation. Most papers get evaluated on a subset of the data sets provided earlier in this report and, for most works, computational efficiency is not an critical aspect of the design. As a result, it was decided to summarize the best performing models on the Cityscapes data set {{cite:b381ed5}}, which has been popular with most real-time architectures as an evaluation benchmark. Table {{table:a4a39d28-2009-4927-a453-81c62e622d3d}}  summarizes the top ten performing models with respect to the mIoU with a short summary of the methods used to achieve these results. Anonymous submissions were not included in this section despite occupying some of the top performing spots in the benchmark evaluation. As can be seen in Table {{table:a4a39d28-2009-4927-a453-81c62e622d3d}} , most entries were published over the past few months, suggesting a very competitive landscape with remarkably fast progress.