MPE: A Mobility Pattern Embedding Model for Predicting Next Locations

Abstract: The wide spread use of positioning and photographing devices gives rise to a deluge of traffic trajectory data (e.g., vehicle passage records and taxi trajectory data), with each record having at least three attributes: object ID, location ID, and time-stamp. In this paper, we propose a novel mobility pattern embedding model called MPE to shed the light on people's mobility patterns in traffic trajectory data from multiple aspects, including sequential, personal, and temporal factors. MPE has two salient features: (1) it is capable of casting various types of information (object, location and time) to an integrated low-dimensional latent space; (2) it considers the effect of ``phantom transitions'' arising from road networks in traffic trajectory data. This embedding model opens the door to a wide range of applications such as next location prediction and visualization. Experimental results on two real-world datasets show that MPE is effective and outperforms the state-of-the-art methods significantly in a variety of tasks.

Destination Prediction by Trajectory Distribution Based Model

Abstract: In this paper we propose a new method to predict the final destination of vehicle trips based on their initial partial trajectories. We first review how we obtained clustering of trajectories that describes user behaviour. Then, we explain how we model main traffic flow patterns by a mixture of 2d Gaussian distributions. This yielded a density based clustering of locations, which produces a data driven grid of similar points within each pattern. We present how this model can be used to predict the final destination of a new trajectory based on their first locations using a two step procedure: We first assign the new trajectory to the clusters it mot likely belongs. Secondly, we use characteristics from trajectories inside these clusters to predict the final destination. Finally, we present experimental results of our methods for classification of trajectories and final destination prediction on datasets of timestamped GPS-Location of taxi trips. We test our methods on two different datasets, to assess the capacity of our method to adapt automatically to different subsets.

A Scalable Framework for Trajectory Prediction

Abstract: Trajectory prediction (TP) is of great importance for a wide range of location-based applications in intelligent transport systems such as location-based advertising, route planning, traffic management, and early warning systems. In the last few years, the widespread use of GPS navigation systems and wireless communication technology enabled vehicles has resulted in huge volumes of trajectory data. The task of utilizing this data employing spatio-temporal techniques for trajectory prediction in an efficient and accurate manner is an ongoing research problem. Existing TP approaches are limited to short-term predictions. Moreover, they cannot handle a large volume of trajectory data for long-term prediction. To address these limitations, we propose a scalable clustering and Markov chain based hybrid framework, called Traj-clusiVAT-based TP, for both short-term and long-term trajectory prediction, which can handle a large number of overlapping trajectories in a dense road network. Traj-clusiVAT can also determine the number of clusters, which represent different movement behaviours in input trajectory data. In our experiments, we compare our proposed approach with a mixed Markov model (MMM)-based scheme, and a trajectory clustering, NETSCAN-based TP method for both short- and long-term trajectory predictions. We performed our experiments on two real, vehicle trajectory datasets, including a large-scale trajectory dataset consisting of 3.28 million trajectories obtained from 15,061 taxis in Singapore over a period of one month. Experimental results on two real trajectory datasets show that our proposed approach outperforms the existing approaches in terms of both short- and long-term prediction performances, based on prediction accuracy and distance error (in km).

Destination Prediction Based on Partial Trajectory Data

Abstract: Two-thirds of the people who buy a new car prefer to use a substitute instead of the built-in navigation system. However, for many applications, knowledge about a user's intended destination and route is crucial. For example, suggestions for available parking spots close to the destination can be made or ride-sharing opportunities along the route are facilitated. Our approach predicts probable destinations and routes of a vehicle, based on the most recent partial trajectory and additional contextual data. The approach follows a three-step procedure: First, a $k$-d tree-based space discretization is performed, mapping GPS locations to discrete regions. Secondly, a recurrent neural network is trained to predict the destination based on partial sequences of trajectories. The neural network produces destination scores, signifying the probability of each region being the destination. Finally, the routes to the most probable destinations are calculated. To evaluate the method, we compare multiple neural architectures and present the experimental results of the destination prediction. The experiments are based on two public datasets of non-personalized, timestamped GPS locations of taxi trips. The best performing models were able to predict the destination of a vehicle with a mean error of 1.3 km and 1.43 km respectively.

Representative studies of human mobility and destination prediction.

Over the last two decades, researchers have devoted a great deal of work to destination and human mobility prediction. This section reviews mainly recent data-driven-based studies, while we have selected some representative studies for the earlier studies. In this section, we focus on their methods and corresponding data conditions. To present the above information more intuitively, we summarize the reviewed literature and form Table {{table:8b31c9d8-2ce1-457b-a9c3-c16d29bf9d62}} . Some studies similar to the literature in the table are omitted, such as studies that use similar methods for the same task on the same dataset. In Table {{table:8b31c9d8-2ce1-457b-a9c3-c16d29bf9d62}} , we present the data condition of the study by “Scene",“Data Type" and “Data Scale", while the method is summarized and presented by “Method". Scene indicates the specific scene of the study. In general, the difficulty of prediction varies from scene to scene. For example, since sharing bike mainly serves the last/first kilometer trip, its destination is usually limited to the nearby station of the departure, while the prediction of taxi and vehicle scene is relatively more difficult. Data Type is used to describe the type of data used in the study. Different studies may use data from various sources, such as LBS-based GPS data, Bluetooth data, smart card data, and so on. Here we ignore the collection devices and abstract them into two types, i.e. “ Trajectory data" and “OD-only" data. The difference between the two is whether information about the mobility or trip process is recorded. For example, in addition to the start and end point, GPS data also records the point of the path, so it belongs to trajectory data. The trajectory data is usually more informative for the “OD-only" data is equivalent to containing only the trajectory data's start and end information. Data Scale is used to describe the scale of data. According to the description of the experimental data scale in different studies, we mainly used two presentation ways in Table {{table:8b31c9d8-2ce1-457b-a9c3-c16d29bf9d62}} : the observation period and the average number of records for an individual.

We first pay attention to the data conditions of existing studies. Trajectory data and OD-only data are the core data types for human mobility and destination prediction, of which studies based on trajectory data account for the vast majority. Data such as weather and temperature are also used for activity-based methods, while such data are not discussed much here. Through Table {{table:8b31c9d8-2ce1-457b-a9c3-c16d29bf9d62}}  we can see that regardless of the type of data used, the existing studies are all based on a large number of historical data of individuals under long-term observations. There are some studies that described its data conditions as “sparse data" like {{cite:3993f9a}}, and others like {{cite:f4c1971}} claimed they focused on “early destination prediction" that recognition in destination prediction has not been fully explored. Essentially, They all want to show their data scales are small and it can be seen according to Table {{table:8b31c9d8-2ce1-457b-a9c3-c16d29bf9d62}}  if we compared data scales of them with studies of the same scene. The prediction under the data condition of small scale is usually considered a more difficult task and there are studies to solve the data sparsity problem in destination prediction like{{cite:3d1d2ec}}. However, we note that the data scale of such studies is only relatively small and none of them explains why and how their data is sparse. The data scale of {{cite:3993f9a}} is actually larger than that of {{cite:68e346f}}, but {{cite:68e346f}} did not describe the data condition as sparse data. So the criteria for data sparsity seem to be ambiguous. With further analysis, we found that almost all existing studies considered that the mobility of individuals follows a certain pattern. For example, {{cite:8b3bd91}} indicated it directly in the text. Other studies like {{cite:1394ee9}} mined individual mobility patterns in its methodology, and {{cite:c19c4f1}} excluded individuals with different routing patterns from the target group. Besides, we also found no studies profiled the regularity or randomness of individuals' mobility. Nevertheless, we still summarised some information by analysing the experimental results. First, we found that the method used in some studies like {{cite:139696f}} does not have the ability to predict destinations where individuals lack observations, but the method still works well experimentally under its data condition. Secondly, in analysing the comparative methods studied, we found that very simple regularity-based methods can also achieve good results under their data conditions such as {{cite:4e6b525}}. The above information indicates that under the existing studies' data conditions, the destination or trajectory is almost completely observed and the regularity of the data is exposed. Combining the above analysis and the definition of data predictability in Section , we considered that the reason why such large scale data was needed is to guarantee the regularity of individual data exposed. More specifically, the data scale of each existing study can ensure the data condition is “high predictability" for its scene and task. Data scale is a critical factor affecting the predictability of data, so when data scale decreases, the predictability of the data decreases accordingly. Based on such recognition, we can understand sparse data as data with relatively low predictability. However, the randomness of the data is also a factor that affects predictability. Hence even the data scale that is relatively small in {{cite:6b4cecf}} is enough to create a data condition with high predictability since it is a simple scene and the important information that origin station is used as input in its task. It is worth mentioning that even for existing studies that claim to be performed under sparse, small-scale data conditions, their data conditions are still of high predictability. This point will be developed in the analysis of the existing studies' methods. In addition to large data scales, there are also studies that ensure high predictability in other ways. For example, {{cite:326d8e5}},{{cite:4cab816}}and {{cite:5288713}} only made prediction for specific highly regular individuals. {{cite:1b0bd3b}} combined with other information such as wind direction to ensure high predictability under relatively short-term observation. In summary, almost all existing studies was developed based on highly predictable data conditions.

Next, we focus on the methods used in existing studies. Nearly all method used for human mobility and destination prediction was covered in Table {{table:8b31c9d8-2ce1-457b-a9c3-c16d29bf9d62}} . We can see that the method is dominated by the sequence model like the Markov model, Recurrent Neural Network(RNN) and its variants like Long Short-Term Memory(LSTM), especially when the data is of the trajectory type. In addition, there are also some studies that used methods based on Bayesian and decision tree and others like clustering. Although many methods have been proposed, we found nearly all of them aimed to learn the pattern of the data and based on which to make the prediction. More plainly, they both tend to reproduce historical or training sequences. So whichever method is used, they treat the data in the same way that is not essentially different from matching directly based on historical trajectories like early-stage study {{cite:e53614a}}, as well as explicitly giving a higher probability to destinations that have historically occurred. Based on such analysis, we can explain why data with high predictability is needed in existing studies for having a stable pattern is a fundamental requirement of these methods for data. On the other hand, almost all these studies seek higher prediction accuracy. Hence, we can summarize the tasks of the existing study as methodological study based on data with high predictability to obtain better or even close to the limit of predictability. And {{cite:6719586}} was a discussion of the upper limit of predictability under highly predictable data conditions.


A Variational Perspective on Diffusion-Based Generative Models and Score Matching

Abstract: Discrete-time diffusion-based generative models and score matching methods have shown promising results in modeling high-dimensional image data. Recently, Song et al. (2021) show that diffusion processes that transform data into noise can be reversed via learning the score function, i.e. the gradient of the log-density of the perturbed data. They propose to plug the learned score function into an inverse formula to define a generative diffusion process. Despite the empirical success, a theoretical underpinning of this procedure is still lacking. In this work, we approach the (continuous-time) generative diffusion directly and derive a variational framework for likelihood estimation, which includes continuous-time normalizing flows as a special case, and can be seen as an infinitely deep variational autoencoder. Under this framework, we show that minimizing the score-matching loss is equivalent to maximizing a lower bound of the likelihood of the plug-in reverse SDE proposed by Song et al. (2021), bridging the theoretical gap.

Score-based Generative Modeling in Latent Space

Abstract: Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. However, they are usually applied directly in data space and often require thousands of network evaluations for sampling. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space, relying on the variational autoencoder framework. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling. To enable training LSGMs end-to-end in a scalable and stable manner, we (i) introduce a new score-matching objective suitable to the LSGM setting, (ii) propose a novel parameterization of the score function that allows SGM to focus on the mismatch of the target distribution with respect to a simple Normal one, and (iii) analytically derive multiple techniques for variance reduction of the training objective. LSGM obtains a state-of-the-art FID score of 2.10 on CIFAR-10, outperforming all existing generative results on this dataset. On CelebA-HQ-256, LSGM is on a par with previous SGMs in sample quality while outperforming them in sampling time by two orders of magnitude. In modeling binary images, LSGM achieves state-of-the-art likelihood on the binarized OMNIGLOT dataset. Our project page and code can be found at https://nvlabs.github.io/LSGM .

Diffusion-GAN: Training GANs with Diffusion

Abstract: Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective in practice. In this paper, we propose Diffusion-GAN, a novel GAN framework that leverages a forward diffusion chain to generate Gaussian-mixture distributed instance noise. Diffusion-GAN consists of three components, including an adaptive diffusion process, a diffusion timestep-dependent discriminator, and a generator. Both the observed and generated data are diffused by the same adaptive diffusion process. At each diffusion timestep, there is a different noise-to-data ratio and the timestep-dependent discriminator learns to distinguish the diffused real data from the diffused generated data. The generator learns from the discriminator's feedback by backpropagating through the forward diffusion chain, whose length is adaptively adjusted to balance the noise and data levels. We theoretically show that the discriminator's timestep-dependent strategy gives consistent and helpful guidance to the generator, enabling it to match the true data distribution. We demonstrate the advantages of Diffusion-GAN over strong GAN baselines on various datasets, showing that it can produce more realistic images with higher stability and data efficiency than state-of-the-art GANs.

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Abstract: A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously addressing three key requirements including: high sample quality, mode coverage, and fast sampling. We call the challenge imposed by these requirements the generative learning trilemma, as the existing models often trade some of them for others. Particularly, denoising diffusion models have shown impressive sample quality and diversity, but their expensive sampling does not yet allow them to be applied in many real-world applications. In this paper, we argue that slow sampling in these models is fundamentally attributed to the Gaussian assumption in the denoising step which is justified only for small step sizes. To enable denoising with large steps, and hence, to reduce the total number of denoising steps, we propose to model the denoising distribution using a complex multimodal distribution. We introduce denoising diffusion generative adversarial networks (denoising diffusion GANs) that model each denoising step using a multimodal conditional GAN. Through extensive evaluations, we show that denoising diffusion GANs obtain sample quality and diversity competitive with original diffusion models while being 2000$\times$ faster on the CIFAR-10 dataset. Compared to traditional GANs, our model exhibits better mode coverage and sample diversity. To the best of our knowledge, denoising diffusion GAN is the first model that reduces sampling cost in diffusion models to an extent that allows them to be applied to real-world applications inexpensively. Project page and code can be found at https://nvlabs.github.io/denoising-diffusion-gan

Diffusion Normalizing Flow

Abstract: We present a novel generative modeling method called diffusion normalizing flow based on stochastic differential equations (SDEs). The algorithm consists of two neural SDEs: a forward SDE that gradually adds noise to the data to transform the data into Gaussian random noise, and a backward SDE that gradually removes the noise to sample from the data distribution. By jointly training the two neural SDEs to minimize a common cost function that quantifies the difference between the two, the backward SDE converges to a diffusion process the starts with a Gaussian distribution and ends with the desired data distribution. Our method is closely related to normalizing flow and diffusion probabilistic models and can be viewed as a combination of the two. Compared with normalizing flow, diffusion normalizing flow is able to learn distributions with sharp boundaries. Compared with diffusion probabilistic models, diffusion normalizing flow requires fewer discretization steps and thus has better sampling efficiency. Our algorithm demonstrates competitive performance in both high-dimension data density estimation and image generation tasks.

Interpreting diffusion score matching using normalizing flow

Abstract: Scoring matching (SM), and its related counterpart, Stein discrepancy (SD) have achieved great success in model training and evaluations. However, recent research shows their limitations when dealing with certain types of distributions. One possible fix is incorporating the original score matching (or Stein discrepancy) with a diffusion matrix, which is called diffusion score matching (DSM) (or diffusion Stein discrepancy (DSD)). However, the lack of interpretation of the diffusion limits its usage within simple distributions and manually chosen matrix. In this work, we plan to fill this gap by interpreting the diffusion matrix using normalizing flows. Specifically, we theoretically prove that DSM (or DSD) is equivalent to the original score matching (or Stein discrepancy) evaluated in the transformed space defined by the normalizing flow, where the diffusion matrix is the inverse of the flow's Jacobian matrix. In addition, we also build its connection to Riemannian manifolds and further extend it to continuous flows, where the change of DSM is characterized by an ODE.

Autoregressive Diffusion Models

Abstract: We introduce Autoregressive Diffusion Models (ARDMs), a model class encompassing and generalizing order-agnostic autoregressive models (Uria et al., 2014) and absorbing discrete diffusion (Austin et al., 2021), which we show are special cases of ARDMs under mild assumptions. ARDMs are simple to implement and easy to train. Unlike standard ARMs, they do not require causal masking of model representations, and can be trained using an efficient objective similar to modern probabilistic diffusion models that scales favourably to highly-dimensional data. At test time, ARDMs support parallel generation which can be adapted to fit any given generation budget. We find that ARDMs require significantly fewer steps than discrete diffusion models to attain the same performance. Finally, we apply ARDMs to lossless compression, and show that they are uniquely suited to this task. Contrary to existing approaches based on bits-back coding, ARDMs obtain compelling results not only on complete datasets, but also on compressing single data points. Moreover, this can be done using a modest number of network calls for (de)compression due to the model's adaptable parallel generation.

Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting

Abstract: In this work, we propose \texttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this end, we use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest through a Markov chain using Langevin sampling. We demonstrate experimentally that the proposed autoregressive denoising diffusion model is the new state-of-the-art multivariate probabilistic forecasting method on real-world data sets with thousands of correlated dimensions. We hope that this method is a useful tool for practitioners and lays the foundation for future research in this area.

Learning Energy-Based Models by Diffusion Recovery Likelihood

Abstract: While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging. Inspired by recent progress on diffusion probabilistic models, we present a diffusion recovery likelihood method to tractably learn and sample from a sequence of EBMs trained on increasingly noisy versions of a dataset. Each EBM is trained with recovery likelihood, which maximizes the conditional probability of the data at a certain noise level given their noisy versions at a higher noise level. Optimizing recovery likelihood is more tractable than marginal likelihood, as sampling from the conditional distributions is much easier than sampling from the marginal distributions. After training, synthesized images can be generated by the sampling process that initializes from Gaussian white noise distribution and progressively samples the conditional distributions at decreasingly lower noise levels. Our method generates high fidelity samples on various image datasets. On unconditional CIFAR-10 our method achieves FID 9.58 and inception score 8.30, superior to the majority of GANs. Moreover, we demonstrate that unlike previous work on EBMs, our long-run MCMC samples from the conditional distributions do not diverge and still represent realistic images, allowing us to accurately estimate the normalized density of data even for high-dimensional datasets. Our implementation is available at https://github.com/ruiqigao/recovery_likelihood.

Latent Diffusion Energy-Based Model for Interpretable Text Modeling

Abstract: Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in generative modeling. Fueled by its flexibility in the formulation and strong modeling power of the latent space, recent works built upon it have made interesting attempts aiming at the interpretability of text modeling. However, latent space EBMs also inherit some flaws from EBMs in data space; the degenerate MCMC sampling quality in practice can lead to poor generation quality and instability in training, especially on data with complex latent structures. Inspired by the recent efforts that leverage diffusion recovery likelihood learning as a cure for the sampling issue, we introduce a novel symbiosis between the diffusion models and latent space EBMs in a variational learning framework, coined as the latent diffusion energy-based model. We develop a geometric clustering-based regularization jointly with the information bottleneck to further improve the quality of the learned latent space. Experiments on several challenging tasks demonstrate the superior performance of our model on interpretable text modeling over strong counterparts.

Diffusion models are incorporated into different generative models.

Connections with Other Generative Models

Generative models are a hot research area that has many applications. For example, they can be utilized to generate high-quality images {{cite:c3ec534}}, synthesize realistic speech and music {{cite:c692b86}},
advance semi-supervised learning {{cite:dfca164}}, {{cite:ab0263e}}, identify adversarial examples {{cite:6b9adde}}, conduct imitation learning {{cite:a1c4a1d}}, and make optimization in reinforcement learning {{cite:70eaa44}}.
In each subsection, we first introduce five other important classes of generative models and analyze their advantages and limitations. Then we introduce how diffusion models are connected with them, and illustrate how these generative models are promoted by incorporating diffusion models. The algorithms that integrate diffusion models with other generative models are summarized in Table {{table:b76cad88-d6be-471a-9467-1874234800eb}} , and we also provide a schematic illustration in Figure {{figure:1f1b8d54-4f76-46ae-b84d-b89cece8f224}} .


Rethinking Atrous Convolution for Semantic Image Segmentation

Abstract: In this work, we revisit atrous convolution, a powerful tool to explicitly adjust filter's field-of-view as well as control the resolution of feature responses computed by Deep Convolutional Neural Networks, in the application of semantic image segmentation. To handle the problem of segmenting objects at multiple scales, we design modules which employ atrous convolution in cascade or in parallel to capture multi-scale context by adopting multiple atrous rates. Furthermore, we propose to augment our previously proposed Atrous Spatial Pyramid Pooling module, which probes convolutional features at multiple scales, with image-level features encoding global context and further boost performance. We also elaborate on implementation details and share our experience on training our system. The proposed `DeepLabv3' system significantly improves over our previous DeepLab versions without DenseCRF post-processing and attains comparable performance with other state-of-art models on the PASCAL VOC 2012 semantic image segmentation benchmark.

Feature Pyramid Networks for Object Detection

Abstract: Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But pyramid representations have been avoided in recent object detectors that are based on deep convolutional networks, partially because they are slow to compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

Pyramid Scene Parsing Network

Abstract: Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed pyramid scene parsing network (PSPNet). Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction. The proposed approach achieves state-of-the-art performance on various datasets. It came first in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark and Cityscapes benchmark. A single PSPNet yields the new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes.

Pyramid Scene Parsing Network

Pyramid Scene Parsing Network

Pyramid Scene Parsing Network

Pyramid Scene Parsing Network

Pyramid Scene Parsing Network

Single-scale testing performance comparison with different baseline methods on NYUDv2 test set.

Our proposed DCA and EDCA are general modules for RGB-D semantic segmentation, which can be easily plugged into CNNs
as attention modules in semantic segmentation. Our method is also assessed with respect to several
representative semantic segmentation architectures: Deeplabv3+ {{cite:1a8fd86}}, Deeplabv3 {{cite:10ed5d4}},
PSPNet {{cite:37aac71}} and FPN {{cite:e151d7a}} with different backbones (ResNet-50, ResNet-101 {{cite:8bdf597}})
on NYUDv2 dataset to verify the generalizability. As is shown in Tab. {{table:d48f34c8-caae-4246-884c-7eadae337555}} , our method outperforms the baseline
by a desirable margin under all settings, demonstrating the generalization capability of our method.


Addressing Fairness in Classification with a Model-Agnostic Multi-Objective Algorithm

Abstract: The goal of fairness in classification is to learn a classifier that does not discriminate against groups of individuals based on sensitive attributes, such as race and gender. One approach to designing fair algorithms is to use relaxations of fairness notions as regularization terms or in a constrained optimization problem. We observe that the hyperbolic tangent function can approximate the indicator function. We leverage this property to define a differentiable relaxation that approximates fairness notions provably better than existing relaxations. In addition, we propose a model-agnostic multi-objective architecture that can simultaneously optimize for multiple fairness notions and multiple sensitive attributes and supports all statistical parity-based notions of fairness. We use our relaxation with the multi-objective architecture to learn fair classifiers. Experiments on public datasets show that our method suffers a significantly lower loss of accuracy than current debiasing algorithms relative to the unconstrained model.

Fairness-Aware Online Meta-learning

Abstract: In contrast to offline working fashions, two research paradigms are devised for online learning: (1) Online Meta Learning (OML) learns good priors over model parameters (or learning to learn) in a sequential setting where tasks are revealed one after another. Although it provides a sub-linear regret bound, such techniques completely ignore the importance of learning with fairness which is a significant hallmark of human intelligence. (2) Online Fairness-Aware Learning. This setting captures many classification problems for which fairness is a concern. But it aims to attain zero-shot generalization without any task-specific adaptation. This therefore limits the capability of a model to adapt onto newly arrived data. To overcome such issues and bridge the gap, in this paper for the first time we proposed a novel online meta-learning algorithm, namely FFML, which is under the setting of unfairness prevention. The key part of FFML is to learn good priors of an online fair classification model's primal and dual parameters that are associated with the model's accuracy and fairness, respectively. The problem is formulated in the form of a bi-level convex-concave optimization. Theoretic analysis provides sub-linear upper bounds for loss regret and for violation of cumulative fairness constraints. Our experiments demonstrate the versatility of FFML by applying it to classification on three real-world datasets and show substantial improvements over the best prior work on the tradeoff between fairness and classification accuracy

Attributing Fair Decisions with Attention Interventions

Abstract: The widespread use of Artificial Intelligence (AI) in consequential domains, such as health-care and parole decision-making systems, has drawn intense scrutiny on the fairness of these methods. However, ensuring fairness is often insufficient as the rationale for a contentious decision needs to be audited, understood, and defended. We propose that the attention mechanism can be used to ensure fair outcomes while simultaneously providing feature attributions to account for how a decision was made. Toward this goal, we design an attention-based model that can be leveraged as an attribution framework. It can identify features responsible for both performance and fairness of the model through attention interventions and attention weight manipulation. Using this attribution framework, we then design a post-processing bias mitigation strategy and compare it with a suite of baselines. We demonstrate the versatility of our approach by conducting experiments on two distinct data types, tabular and textual.

Publications that benchmark with approaches that are not a bias mitigation method.

Table {{table:5f5f683c-5c7f-41d7-8053-165321d99983}}  shows the publications that benchmark their proposed method against non-bias mitigation methods, according to the type of approach applied.
Among the 12 types of approaches, as shown in Section REF  - REF , seven can be found to benchmark against non-bias methods.
This occurs rarely for post-processing methods, three publications in total, with one per approach type.
14 and 18 publications for pre-processing and in-processing methods respectively, benchmark against non-bias mitigation methods.


A Seed-based Plant Propagation Algorithm: The Feeding Station Model

Abstract: The seasonal production of fruit and seeds resembles opening a feeding station, such as a restaurant agents/ customers will arrive at a certain rate and pick fruit (get served) at a certain rate following some appropriate processes. Therefore, dispersion follows the resource process. Modelling this process results in a search/optimisation algorithm that used dispersion as an exploration tool that, if well captured, will find the optimum of a function over a given search space. This paper presents such an algorithm and tests it on non-trivial problems.

A Numerical Optimization Algorithm Inspired by the Strawberry Plant

Abstract: This paper proposes a new numerical optimization algorithm inspired by the strawberry plant for solving complicated engineering problems. Plants like strawberry develop both runners and roots for propagation and search for water resources and minerals. In these plants, runners and roots can be thought of as tools for global and local searches, respectively. The proposed algorithm has three main differences with the trivial nature-inspired optimization algorithms: duplication-elimination of the computational agents at all iterations, subjecting all agents to both small and large movements from the beginning to end, and the lack of communication (information exchange) between agents. Moreover, it has the advantage of using only three parameters to be tuned by user. This algorithm is applied to standard test functions and the results are compared with GA and PSO. The proposed algorithm is also used to solve an open problem in the field of robust control theory. These simulations show that the proposed algorithm can very effectively solve complicated optimization problems.

Algorithms based on plants behavior.

Growing of plants as an adaptive phenomenon is represented in several algorithms like transmission mode of bean seeds ({{cite:8f4eff5}}). The growing process of trees where branches growth toward betters positions by phototropism and photosynthesis ({{cite:69711c8}}). In a more sophisticated way, a complete process of a forest growth has been represented where trees can live more time in specific zones and others trees can not ({{cite:08177e4}}). Some Seeds falls near of these trees but others are taken to another promising zones. A similar process where an artificial plant growth is represented including growth of leaves and spatial branching by phototropism ({{cite:4641201}}). Strawberry plant which try to find better conditions of water, light and nutrients investing more in promising near neighbor spots has inspired some algorithms ({{cite:d7900dc}}). The colonization in a cropping of invasive weeds ({{cite:5de2ff2}}). Seeds in more fertile soils, grow more and produce more seeds on paddy fields ({{cite:aa1ae8c}}). Other mechanisms used by plants to defend against different predators since fungus, to insects or herbivores ({{cite:286ea8b}}). Pollination process of the flowers represents different strategies followed by the plants to increase its reproduction ({{cite:08177e4}}), and the growth of roots to search high nutrients ({{cite:86d73d5}}, {{cite:beec593}}). A representation of the growth of root tree for searching wet soil ({{cite:3828f2d}}). All algorithms based on plants mechanisms are shown in Table  {{table:8279cc19-a443-45f7-a438-7d0ded00195e}}


A Microservice-enabled Architecture for Smart Surveillance using Blockchain Technology

Abstract: While the smart surveillance system enhanced by the Internet of Things (IoT) technology becomes an essential part of Smart Cities, it also brings new concerns in security of the data. Compared to the traditional surveillance systems that is built following a monolithic architecture to carry out lower level operations, such as monitoring and recording, the modern surveillance systems are expected to support more scalable and decentralized solutions for advanced video stream analysis at the large volumes of distributed edge devices. In addition, the centralized architecture of the conventional surveillance systems is vulnerable to single point of failure and privacy breach owning to the lack of protection to the surveillance feed. This position paper introduces a novel secure smart surveillance system based on microservices architecture and blockchain technology. Encapsulating the video analysis algorithms as various independent microservices not only isolates the video feed from different sectors, but also improve the system availability and robustness by decentralizing the operations. The blockchain technology securely synchronizes the video analysis databases among microservices across surveillance domains, and provides tamper proof of data in the trustless network environment. Smart contract enabled access authorization strategy prevents any unauthorized user from accessing the microservices and offers a scalable, decentralized and fine-grained access control solution for smart surveillance systems.

Building Critical Applications Using Microservices

Abstract: Safeguarding the correctness of critical software is a grand challenge. A microservice-based system is described that builds trustworthy systems on top of legacy hardware and software components, ensuring microservices' integrity, confidentiality, and correct execution with the help of secure enclaves.

Microservices API Security

Abstract: —Microservices have become the hot term since 2014. This particular programming methodology involves an architectural approach that emphasizes the decomposition of applications into single purpose, with implementation via the Rest API. A new software development style that has grown from recent trends in software development/management practices, Microservices are composable, fine-grained modules which can be independently deployable and scaled both vertically and horizontally. This empowers the development, manageability and speed to the changing market by adapting to Agile methods, DevOps culture, cloud, Linux containers, and Continuous Integration/ Continuous Development methods. This paper will focus on the necessity for this methodology as well as it’s corresponding API security options needed for securing all of the back-end Microservices

List of selected studies: C: Conference paper, J: Journal paper, A: Automatic search, S: Snowballing

Table {{table:279a985d-191e-468a-acfa-b757e0360f3b}}  shows the complete list of selected studies with their year and type of publication and how they are found.


Sparse Autoencoder for Unsupervised Nucleus Detection and Representation in Histopathology Images

Abstract: Histopathology images are crucial to the study of complex diseases such as cancer. The histologic characteristics of nuclei play a key role in disease diagnosis, prognosis and analysis. In this work, we propose a sparse Convolutional Autoencoder (CAE) for fully unsupervised, simultaneous nucleus detection and feature extraction in histopathology tissue images. Our CAE detects and encodes nuclei in image patches in tissue images into sparse feature maps that encode both the location and appearance of nuclei. Our CAE is the first unsupervised detection network for computer vision applications. The pretrained nucleus detection and feature extraction modules in our CAE can be fine-tuned for supervised learning in an end-to-end fashion. We evaluate our method on four datasets and reduce the errors of state-of-the-art methods up to 42%. We are able to achieve comparable performance with only 5% of the fully-supervised annotation cost.

Classification of Histopathological Biopsy Images Using Ensemble of Deep Learning Networks

Abstract: Breast cancer is one of the leading causes of death across the world in women. Early diagnosis of this type of cancer is critical for treatment and patient care. Computer-aided detection (CAD) systems using convolutional neural networks (CNN) could assist in the classification of abnormalities. In this study, we proposed an ensemble deep learning-based approach for automatic binary classification of breast histology images. The proposed ensemble model adapts three pre-trained CNNs, namely VGG19, MobileNet, and DenseNet. The ensemble model is used for the feature representation and extraction steps. The extracted features are then fed into a multi-layer perceptron classifier to carry out the classification task. Various pre-processing and CNN tuning techniques such as stain-normalization, data augmentation, hyperparameter tuning, and fine-tuning are used to train the model. The proposed method is validated on four publicly available benchmark datasets, i.e., ICIAR, BreakHis, PatchCamelyon, and Bioimaging. The proposed multi-model ensemble method obtains better predictions than single classifiers and machine learning algorithms with accuracies of 98.13%, 95.00%, 94.64% and 83.10% for BreakHis, ICIAR, PatchCamelyon and Bioimaging datasets, respectively.

Summary of publicly available HI datasets.

Given the importance of datasets for the research on HI, we have also compiled in Tables {{table:9a0bd20a-ddb2-4dec-9ebd-d266e172d626}}  and {{table:4262b91b-cf7f-49a8-a651-47f43449b7d4}} , a list of the datasets that have been used in experiments of several works we covered in this review. We included the dataset reference, year of creation, their contents in terms of the number of images and patients, and references to the papers that have used them.

Finally, Tables {{table:9a0bd20a-ddb2-4dec-9ebd-d266e172d626}}  and {{table:4262b91b-cf7f-49a8-a651-47f43449b7d4}}  also help us to understand the increasing interest in HI analysis in the last years. We have found that most of the early works are based on small private datasets, what makes difficult for other researchers that do not have access to such HI datasets to carry out research in this area as well as to reproduce the scientific results. On the other hand, most of the recent works are based on public HI datasets, which are a great contribution to the science as they provide a way to researchers to develop new methods and compare their performance with the existing ones. However, there is still a lack of large scale supervised WSI datasets.


End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Abstract: State-of-the-art sequence labeling systems traditionally require large amounts of task-specific knowledge in the form of hand-crafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both word- and character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF. Our system is truly end-to-end, requiring no feature engineering or data pre-processing, thus making it applicable to a wide range of sequence labeling tasks. We evaluate our system on two data sets for two sequence labeling tasks --- Penn Treebank WSJ corpus for part-of-speech (POS) tagging and CoNLL 2003 corpus for named entity recognition (NER). We obtain state-of-the-art performance on both the two data --- 97.55\% accuracy for POS tagging and 91.21\% F1 for NER.

Bidirectional LSTM-CRF Models for Sequence Tagging

Abstract: In this paper, we propose a variety of Long Short-Term Memory (LSTM) based models for sequence tagging. These models include LSTM networks, bidirectional LSTM (BI-LSTM) networks, LSTM with a Conditional Random Field (CRF) layer (LSTM-CRF) and bidirectional LSTM with a CRF layer (BI-LSTM-CRF). Our work is the first to apply a bidirectional LSTM CRF (denoted as BI-LSTM-CRF) model to NLP benchmark sequence tagging data sets. We show that the BI-LSTM-CRF model can efficiently use both past and future input features thanks to a bidirectional LSTM component. It can also use sentence level tag information thanks to a CRF layer. The BI-LSTM-CRF model can produce state of the art (or close to) accuracy on POS, chunking and NER data sets. In addition, it is robust and has less dependence on word embedding as compared to previous observations.

English NER Studies

Neural Models for Named-Entity Recognition

Earlier traditional named entity recognition systems have relied heavily on feature engineering and employed hand-crafted language dependent features, large gazetteers, and tagged datasets {{cite:416a194}}. A significant branch of research has utilized a range of statistical approaches to address the problem such as maximum entropy classifiers {{cite:cbef4a4}}, decision trees {{cite:da9bbde}}, and conditional random fields {{cite:13e2918}}. However, in recent years, the focus of NER research has shifted to neural models in parallel with observed improvements on multiple language processing benchmarks such as question-answering and language generation. Neural NER models have been guided by distributional approaches where the meaning of a word is carried in its surroundings via vector representations {{cite:61119d0}}, {{cite:4ea96b6}}, {{cite:f6631a4}}. Initial attempts considered words as separate tokens and represented each token using a fixed-length vector {{cite:e9f0d70}}, {{cite:affc912}}. Some other studies explored different ways of representing words such as concatenating embeddings of characters {{cite:59a197e}}, morphemes {{cite:bce0b4e}}, or other word subparts to fixed-length word embeddings. In recent NER studies, the problem was formulated as a sequence labeling task and different Seq2Seq models were shown to achieve state-of-the art results where final embeddings of words are encoded by gated recurrent units (GRUs) or long short term memory units (LSTMs) {{cite:6cc030c}}, {{cite:89da900}}, {{cite:be55488}}. Using conditional random fields on top of neural networks were proved to work equally well {{cite:416a194}}, {{cite:d70f4ef}}, {{cite:9a0f8d3}} or better than previous methods. Moreover, BiLSTM-CRF models with character and word embeddings were shown to be effective for multiple languages including Chinese {{cite:21569a3}} and arguably considered as a base model for tagging {{cite:f96eb8a}}. Unfortunately, a word, no matter in which context it appears, is represented with the same final embedding in these models. A recent study has utilized contextual string embeddings {{cite:86895a6}}, where final word embeddings are contextualized according to the entire sentence. In that study, all characters in the sentence up to the last character of a word were processed via a forward LSTM and all characters from the end to the beginning of the sentence were processed via a backward LSTM. The obtained hidden states were then concatenated to produce final embedding of the word in focus. This kind of word embeddings was proved to improve not only NER tagging but also other sequence labeling tasks such as part-of-speech labeling and phrase chunking. Achieved performance scores (f1 scores) of some these English NER studies are given in Table {{table:91f14610-b576-455d-8eca-078df81209d6}} .


Deep Placental Vessel Segmentation for Fetoscopic Mosaicking

Abstract: During fetoscopic laser photocoagulation, a treatment for twin-to-twin transfusion syndrome (TTTS), the clinician first identifies abnormal placental vascular connections and laser ablates them to regulate blood flow in both fetuses. The procedure is challenging due to the mobility of the environment, poor visibility in amniotic fluid, occasional bleeding, and limitations in the fetoscopic field-of-view and image quality. Ideally, anastomotic placental vessels would be automatically identified, segmented and registered to create expanded vessel maps to guide laser ablation, however, such methods have yet to be clinically adopted. We propose a solution utilising the U-Net architecture for performing placental vessel segmentation in fetoscopic videos. The obtained vessel probability maps provide sufficient cues for mosaicking alignment by registering consecutive vessel maps using the direct intensity-based technique. Experiments on 6 different in vivo fetoscopic videos demonstrate that the vessel intensity-based registration outperformed image intensity-based registration approaches showing better robustness in qualitative and quantitative comparison. We additionally reduce drift accumulation to negligible even for sequences with up to 400 frames and we incorporate a scheme for quantifying drift error in the absence of the ground-truth. Our paper provides a benchmark for fetoscopy placental vessel segmentation and registration by contributing the first in vivo vessel segmentation and fetoscopic videos dataset.

Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints

Abstract: Estimating a scene reconstruction and the camera motion from in-body videos is challenging due to several factors, e.g. the deformation of in-body cavities or the lack of texture. In this paper we present Endo-Depth-and-Motion, a pipeline that estimates the 6-degrees-of-freedom camera pose and dense 3D scene models from monocular endoscopic videos. Our approach leverages recent advances in self-supervised depth networks to generate pseudo-RGBD frames, then tracks the camera pose using photometric residuals and fuses the registered depth maps in a volumetric representation. We present an extensive experimental evaluation in the public dataset Hamlyn, showing high-quality results and comparisons against relevant baselines. We also release all models and code for future comparisons.

Stereo Correspondence and Reconstruction of Endoscopic Data Challenge

Abstract: The stereo correspondence and reconstruction of endoscopic data sub-challenge was organized during the Endovis challenge at MICCAI 2019 in Shenzhen, China. The task was to perform dense depth estimation using 7 training datasets and 2 test sets of structured light data captured using porcine cadavers. These were provided by a team at Intuitive Surgical. 10 teams participated in the challenge day. This paper contains 3 additional methods which were submitted after the challenge finished as well as a supplemental section from these teams on issues they found with the dataset.

SERV-CT: A disparity dataset from CT for validation of endoscopic 3D reconstruction

Abstract: In computer vision, reference datasets have been highly successful in promoting algorithmic development in stereo reconstruction. Surgical scenes gives rise to specific problems, including the lack of clear corner features, highly specular surfaces and the presence of blood and smoke. Publicly available datasets have been produced using CT and either phantom images or biological tissue samples covering a relatively small region of the endoscope field-of-view. We present a stereo-endoscopic reconstruction validation dataset based on CT (SERV-CT). Two {\it ex vivo} small porcine full torso cadavers were placed within the view of the endoscope with both the endoscope and target anatomy visible in the CT scan. Orientation of the endoscope was manually aligned to the stereoscopic view. Reference disparities and occlusions were calculated for 8 stereo pairs from each sample. For the second sample an RGB surface was acquired to aid alignment of smooth, featureless surfaces. Repeated manual alignments showed an RMS disparity accuracy of ~2 pixels and a depth accuracy of ~2mm. The reference dataset includes endoscope image pairs with corresponding calibration, disparities, depths and occlusions covering the majority of the endoscopic image and a range of tissue types. Smooth specular surfaces and images with significant variation of depth are included. We assessed the performance of various stereo algorithms from online available repositories. There is a significant variation between algorithms, highlighting some of the challenges of surgical endoscopic images. The SERV-CT dataset provides an easy to use stereoscopic validation for surgical applications with smooth reference disparities and depths with coverage over the majority of the endoscopic images. This complements existing resources well and we hope will aid the development of surgical endoscopic anatomical reconstruction algorithms.

Qualitative Comparison of Image Stitching Algorithms for Multi-Camera Systems in Laparoscopy

Abstract: Multi-camera systems were recently introduced into laparoscopy to increase the narrow field of view of the surgeon. The video streams are stitched together to create a panorama that is easier for the surgeon to comprehend. Multi-camera prototypes for laparoscopy use quite basic algorithms and have only been evaluated on simple laparoscopic scenarios. The more recent state-of-the-art algorithms, mainly designed for the smartphone industry, have not yet been evaluated in laparoscopic conditions. We developed a simulated environment to generate a dataset of multi-view images displaying a wide range of laparoscopic situations, which is adaptable to any multi-camera system. We evaluated classical and state-of-the-art image stitching techniques used in non-medical applications on this dataset, including one unsupervised deep learning approach. We show that classical techniques that use global homography fail to provide a clinically satisfactory rendering and that even the most recent techniques, despite providing high quality panorama images in non-medical situations, may suffer from poor alignment or severe distortions in simulated laparoscopic scenarios. We highlight the main advantages and flaws of each algorithm within a laparoscopic context, identify the main remaining challenges that are specific to laparoscopy, and propose methods to improve these approaches. We provide public access to the simulated environment and dataset.

Bimodal Camera Pose Prediction for Endoscopy

Abstract: Deducing the 3D structure of endoscopic scenes from images is exceedingly challenging. In addition to deformation and view-dependent lighting, tubular structures like the colon present problems stemming from their self-occluding and repetitive anatomical structure. In this paper, we propose SimCol, a synthetic dataset for camera pose estimation in colonoscopy, and a novel method that explicitly learns a bimodal distribution to predict the endoscope pose. Our dataset replicates real colonoscope motion and highlights the drawbacks of existing methods. We publish 18k RGB images from simulated colonoscopy with corresponding depth and camera poses and make our data generation environment in Unity publicly available. We evaluate different camera pose prediction methods and demonstrate that, when trained on our data, they generalize to real colonoscopy sequences, and our bimodal approach outperforms prior unimodal work. Our project and dataset can be found here: https://www.github.com/anitarau/simcol.

EndoMapper dataset of complete calibrated endoscopy procedures

Abstract: Computer-assisted systems are becoming broadly used in medicine. In endoscopy, most research focuses on the automatic detection of polyps or other pathologies, but localization and navigation of the endoscope are completely performed manually by physicians. To broaden this research and bring spatial Artificial Intelligence to endoscopies, data from complete procedures is needed. This paper introduces the Endomapper dataset, the first collection of complete endoscopy sequences acquired during regular medical practice, making secondary use of medical data. Its main purpose is to facilitate the development and evaluation of Visual Simultaneous Localization and Mapping (VSLAM) methods in real endoscopy data. The dataset contains more than 24 hours of video. It is the first endoscopic dataset that includes endoscope calibration as well as the original calibration videos. Meta-data and annotations associated with the dataset vary from the anatomical landmarks, procedure labeling, segmentations, reconstructions, simulated sequences with ground truth and same patient procedures. The software used in this paper is publicly available.

Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration

Abstract: Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a Colonoscopy 3D Video Dataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at durr.jhu.edu/C3VD.

SurgT challenge: Benchmark of Soft-Tissue Trackers for Robotic Surgery

Abstract: This paper introduces the ``SurgT: Surgical Tracking" challenge which was organised in conjunction with MICCAI 2022. There were two purposes for the creation of this challenge: (1) the establishment of the first standardised benchmark for the research community to assess soft-tissue trackers; and (2) to encourage the development of unsupervised deep learning methods, given the lack of annotated data in surgery. A dataset of 157 stereo endoscopic videos from 20 clinical cases, along with stereo camera calibration parameters, have been provided. Participants were assigned the task of developing algorithms to track the movement of soft tissues, represented by bounding boxes, in stereo endoscopic videos. At the end of the challenge, the developed methods were assessed on a previously hidden test subset. This assessment uses benchmarking metrics that were purposely developed for this challenge, to verify the efficacy of unsupervised deep learning algorithms in tracking soft-tissue. The metric used for ranking the methods was the Expected Average Overlap (EAO) score, which measures the average overlap between a tracker's and the ground truth bounding boxes. Coming first in the challenge was the deep learning submission by ICVS-2Ai with a superior EAO score of 0.617. This method employs ARFlow to estimate unsupervised dense optical flow from cropped images, using photometric and regularization losses. Second, Jmees with an EAO of 0.583, uses deep learning for surgical tool segmentation on top of a non-deep learning baseline method: CSRT. CSRT by itself scores a similar EAO of 0.563. The results from this challenge show that currently, non-deep learning methods are still competitive. The dataset and benchmarking tool created for this challenge have been made publicly available at https://surgt.grand-challenge.org/.

Learning How To Robustly Estimate Camera Pose in Endoscopic Videos

Abstract: Purpose: Surgical scene understanding plays a critical role in the technology stack of tomorrow's intervention-assisting systems in endoscopic surgeries. For this, tracking the endoscope pose is a key component, but remains challenging due to illumination conditions, deforming tissues and the breathing motion of organs. Method: We propose a solution for stereo endoscopes that estimates depth and optical flow to minimize two geometric losses for camera pose estimation. Most importantly, we introduce two learned adaptive per-pixel weight mappings that balance contributions according to the input image content. To do so, we train a Deep Declarative Network to take advantage of the expressiveness of deep-learning and the robustness of a novel geometric-based optimization approach. We validate our approach on the publicly available SCARED dataset and introduce a new in-vivo dataset, StereoMIS, which includes a wider spectrum of typically observed surgical settings. Results: Our method outperforms state-of-the-art methods on average and more importantly, in difficult scenarios where tissue deformations and breathing motion are visible. We observed that our proposed weight mappings attenuate the contribution of pixels on ambiguous regions of the images, such as deforming tissues. Conclusion: We demonstrate the effectiveness of our solution to robustly estimate the camera pose in challenging endoscopic surgical scenes. Our contributions can be used to improve related tasks like simultaneous localization and mapping (SLAM) or 3D reconstruction, therefore advancing surgical scene understanding in minimally-invasive surgery.

Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking

Abstract: Accurate and robust tracking and reconstruction of the surgical scene is a critical enabling technology toward autonomous robotic surgery. Existing algorithms for 3D perception in surgery mainly rely on geometric information, while we propose to also leverage semantic information inferred from the endoscopic video using image segmentation algorithms. In this paper, we present a novel, comprehensive surgical perception framework, Semantic-SuPer, that integrates geometric and semantic information to facilitate data association, 3D reconstruction, and tracking of endoscopic scenes, benefiting downstream tasks like surgical navigation. The proposed framework is demonstrated on challenging endoscopic data with deforming tissue, showing its advantages over our baseline and several other state-of the-art approaches. Our code and dataset are available at https://github.com/ucsdarclab/Python-SuPer.

STIR: Surgical Tattoos in Infrared

Abstract: Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-flourescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in-vivo and ex-vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548

Datasets released with applications in tracking and mapping. Reconstruction (Recon.), Abdomen (Abd.), Fetoscopy (Fet.). Truth abbreviations: Bounding boxes (BB), Visible (Vis), Infrared (IR), Depth Mapping: DM., Transfer learning (Transf.). R/S/P: (Real/Simulated/Phantom).

We will begin in Section REF  with a summary of datasets that do not have any ground truth which are primarily useful for training unsupervised methods.
In Section REF , we delve into datasets with algorithmically generated ground truth.
The algorithmically generated datasets are in their own section because they depend on reconstruction accuracy of stereo algorithms or SfM, and are limited to be at best as good as the classical reconstruction methods.
We will then follow with summarizing simulated ground truth datasets, generated via rendered 3D models, in Section REF , and physical phantoms, e.g., silicone tissue models, in Section REF .
We finally close with ground truth that uses real tissue in Section REF .
We separate these into these classes as different data types can be vulnerable to different biases.
For example, simulation or phantom data might not carry over to real data.
Alongside the sections, we have a table of datasets to reference in Table {{table:20649c3c-ac36-4ef6-a852-c89d5aa88faf}} , and a figure showing their availability over time in Fig. {{figure:0b131658-7ae6-4903-a308-640c533521d7}} .
This table should provide a means to get a high level summary of different algorithmic approaches and dataset generation.
For our discussion on the datasets, please go to Section REF  near the end of this review.

Thus, having a real dataset is important.
Even with this, there are still difficulties.
Many real datasets are rigid as seen in Table {{table:20649c3c-ac36-4ef6-a852-c89d5aa88faf}} .
Even if non-rigid, they can still be different from what we would find clinically.
Just as we cannot know how well synthetic data can be used, the same argument stands in a weaker manner for porcine tissue instead of human tissue.
A similar argument goes for visible markers.
In short the question comes down to, how can we reduce both the bias and difficulty that occurs in generating ground truth.
Some methods use non-medical datasets to evaluate their algorithms {{cite:b9629d1}}.
Concepts such as crowd sourcing, or means to ease the labelling process with markers or IR tattoos are relevant here.
To conclude, more work needs to be done with relevant clinical experiments to know what data we have and need.


Combines FL and fog computing to reduce mitigation time and improve detection
accuracy

* Self- learning Anomaly Detection System for IoT devices.

* Support of low Security Measures.
* Low computation power
* High Communication Latency

Federated learning with hierarchical clustering of local updates to improve training on non-IID data

Abstract: Federated learning (FL) is a well established method for performing machine learning tasks over massively distributed data. However in settings where data is distributed in a non-iid (not independent and identically distributed) fashion -- as is typical in real world situations -- the joint model produced by FL suffers in terms of test set accuracy and/or communication costs compared to training on iid data. We show that learning a single joint model is often not optimal in the presence of certain types of non-iid data. In this work we present a modification to FL by introducing a hierarchical clustering step (FL+HC) to separate clusters of clients by the similarity of their local updates to the global joint model. Once separated, the clusters are trained independently and in parallel on specialised models. We present a robust empirical analysis of the hyperparameters for FL+HC for several iid and non-iid settings. We show how FL+HC allows model training to converge in fewer communication rounds (significantly so under some non-iid settings) compared to FL without clustering. Additionally, FL+HC allows for a greater percentage of clients to reach a target accuracy compared to standard FL. Finally we make suggestions for good default hyperparameters to promote superior performing specialised models without modifying the the underlying federated learning communication protocol.

FLEAM: A Federated Learning Empowered Architecture to Mitigate DDoS in Industrial IoT

Abstract: The distributed denial of service (DDoS) attack is detrimental to the industrial Internet of things (IIoT) as it triggers severe resource starvation on networked objects. Recent dynamics demonstrate that it is a highly profitable business for attackers using botnets. Current centralized mitigation solutions concentrate on detection and mitigation at a victim's side, paying inadequate attention to hacking costs and the collaboration of defenders. Thus, we propose the federated learning empowered mitigation architecture (FLEAM) to advocate joint defense, incurring a higher hacking expense. FLEAM combines FL and fog computing to reduce mitigation time and improve detection accuracy, enabling defenders to jointly combatting botnets. Our comprehensive evaluations showcase that the attacking expense incurred is 2.5 times higher, the mitigation delay is about 72% lower, and the accuracy is 47% greater on average than classic solutions.

D\"IoT: A Federated Self-learning Anomaly Detection System for IoT

Abstract: IoT devices are increasingly deployed in daily life. Many of these devices are, however, vulnerable due to insecure design, implementation, and configuration. As a result, many networks already have vulnerable IoT devices that are easy to compromise. This has led to a new category of malware specifically targeting IoT devices. However, existing intrusion detection techniques are not effective in detecting compromised IoT devices given the massive scale of the problem in terms of the number of different types of devices and manufacturers involved. In this paper, we present D\"IoT, an autonomous self-learning distributed system for detecting compromised IoT devices effectively. In contrast to prior work, D\"IoT uses a novel self-learning approach to classify devices into device types and build normal communication profiles for each of these that can subsequently be used to detect anomalous deviations in communication patterns. D\"IoT utilizes a federated learning approach for aggregating behavior profiles efficiently. To the best of our knowledge, it is the first system to employ a federated learning approach to anomaly-detection-based intrusion detection. Consequently, D\"IoT can cope with emerging new and unknown attacks. We systematically and extensively evaluated more than 30 off-the-shelf IoT devices over a long term and show that D\"IoT is highly effective (95.6% detection rate) and fast (~257 ms) at detecting devices compromised by, for instance, the infamous Mirai malware. D\"IoT reported no false alarms when evaluated in a real-world smart home deployment setting.

Summary of FL literature review for IDS

In this section, the state-of-the-art review on FL for the use of anomaly-based IDS is exhibited. The practicality of FL architecture for the implementation of IDS is explored through key terms such as privacy, communication, data storage, and efficiency. A summary of the same is provided in table {{table:a9df73a8-520e-4fa6-97cb-cdb9370120b8}} .


Semi-Supervised Classification with Graph Convolutional Networks

Abstract: We present a scalable approach for semi-supervised learning on graph-structured data that is based on an efficient variant of convolutional neural networks which operate directly on graphs. We motivate the choice of our convolutional architecture via a localized first-order approximation of spectral graph convolutions. Our model scales linearly in the number of graph edges and learns hidden layer representations that encode both local graph structure and features of nodes. In a number of experiments on citation networks and on a knowledge graph dataset we demonstrate that our approach outperforms related methods by a significant margin.

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

Abstract: \emph{Over-fitting} and \emph{over-smoothing} are two main obstacles of developing deep Graph Convolutional Networks (GCNs) for node classification. In particular, over-fitting weakens the generalization ability on small dataset, while over-smoothing impedes model training by isolating output representations from the input features with the increase in network depth. This paper proposes DropEdge, a novel and flexible technique to alleviate both issues. At its core, DropEdge randomly removes a certain number of edges from the input graph at each training epoch, acting like a data augmenter and also a message passing reducer. Furthermore, we theoretically demonstrate that DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by it. More importantly, our DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance. Extensive experiments on several benchmarks verify that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs. The effect of DropEdge on preventing over-smoothing is empirically visualized and validated as well. Codes are released on~\url{https://github.com/DropEdge/DropEdge}.

Training Robust Graph Neural Networks with Topology Adaptive Edge Dropping

Abstract: Graph neural networks (GNNs) are processing architectures that exploit graph structural information to model representations from network data. Despite their success, GNNs suffer from sub-optimal generalization performance given limited training data, referred to as over-fitting. This paper proposes Topology Adaptive Edge Dropping (TADropEdge) method as an adaptive data augmentation technique to improve generalization performance and learn robust GNN models. We start by explicitly analyzing how random edge dropping increases the data diversity during training, while indicating i.i.d. edge dropping does not account for graph structural information and could result in noisy augmented data degrading performance. To overcome this issue, we consider graph connectivity as the key property that captures graph topology. TADropEdge incorporates this factor into random edge dropping such that the edge-dropped subgraphs maintain similar topology as the underlying graph, yielding more satisfactory data augmentation. In particular, TADropEdge first leverages the graph spectrum to assign proper weights to graph edges, which represent their criticality for establishing the graph connectivity. It then normalizes the edge weights and drops graph edges adaptively based on their normalized weights. Besides improving generalization performance, TADropEdge reduces variance for efficient training and can be applied as a generic method modular to different GNN models. Intensive experiments on real-life and synthetic datasets corroborate theory and verify the effectiveness of the proposed method.

Data Augmentation for Graph Neural Networks

Abstract: Data augmentation has been widely used to improve generalizability of machine learning models. However, comparatively little work studies data augmentation for graphs. This is largely due to the complex, non-Euclidean structure of graphs, which limits possible manipulation operations. Augmentation operations commonly used in vision and language have no analogs for graphs. Our work studies graph data augmentation for graph neural networks (GNNs) in the context of improving semi-supervised node-classification. We discuss practical and theoretical motivations, considerations and strategies for graph data augmentation. Our work shows that neural edge predictors can effectively encode class-homophilic structure to promote intra-class edges and demote inter-class edges in given graph structure, and our main contribution introduces the GAug graph data augmentation framework, which leverages these insights to improve performance in GNN-based node classification via edge prediction. Extensive experiments on multiple benchmarks show that augmentation via GAug improves performance across GNN architectures and datasets.

Local Augmentation for Graph Neural Networks

Abstract: Graph Neural Networks (GNNs) have achieved remarkable performance on graph-based tasks. The key idea for GNNs is to obtain informative representation through aggregating information from local neighborhoods. However, it remains an open question whether the neighborhood information is adequately aggregated for learning representations of nodes with few neighbors. To address this, we propose a simple and efficient data augmentation strategy, local augmentation, to learn the distribution of the node features of the neighbors conditioned on the central node's feature and enhance GNN's expressive power with generated features. Local augmentation is a general framework that can be applied to any GNN model in a plug-and-play manner. It samples feature vectors associated with each node from the learned conditional distribution as additional input for the backbone model at each training iteration. Extensive experiments and analyses show that local augmentation consistently yields performance improvement when applied to various GNN architectures across a diverse set of benchmarks. For example, experiments show that plugging in local augmentation to GCN and GAT improves by an average of 3.4\% and 1.6\% in terms of test accuracy on Cora, Citeseer, and Pubmed. Besides, our experimental results on large graphs (OGB) show that our model consistently improves performance over backbones. Code is available at https://github.com/SongtaoLiu0823/LAGNN.

Graph Random Neural Network for Semi-Supervised Learning on Graphs

Abstract: We study the problem of semi-supervised learning on graphs, for which graph neural networks (GNNs) have been extensively explored. However, most existing GNNs inherently suffer from the limitations of over-smoothing, non-robustness, and weak-generalization when labeled nodes are scarce. In this paper, we propose a simple yet effective framework -- GRAPH RANDOM NEURAL NETWORKS (GRAND) -- to address these issues. In GRAND, we first design a random propagation strategy to perform graph data augmentation. Then we leverage consistency regularization to optimize the prediction consistency of unlabeled nodes across different data augmentations. Extensive experiments on graph benchmark datasets suggest that GRAND significantly outperforms state-of-the-art GNN baselines on semi-supervised node classification. Finally, we show that GRAND mitigates the issues of over-smoothing and non-robustness, exhibiting better generalization behavior than existing GNNs. The source code of GRAND is publicly available at https://github.com/Grand20/grand.

Graph Attention Networks

Abstract: We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).

Inductive Representation Learning on Large Graphs

Abstract: Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches are inherently transductive and do not naturally generalize to unseen nodes. Here we present GraphSAGE, a general, inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data. Instead of training individual embeddings for each node, we learn a function that generates embeddings by sampling and aggregating features from a node's local neighborhood. Our algorithm outperforms strong baselines on three inductive node-classification benchmarks: we classify the category of unseen nodes in evolving information graphs based on citation and Reddit post data, and we show that our algorithm generalizes to completely unseen graphs using a multi-graph dataset of protein-protein interactions.

Node-Level Tasks Data Augmentation Methods Accuracy(%) Comparisons with SOTAs

This section compares the performance of the existing data augmentation methods for node-level tasks against state-of-the-arts (SOTAs) GNN architectures GCN {{cite:120b72c}}, GAT {{cite:97d62ac}}, and GraphSAGE {{cite:7aadc71}}. Table {{table:b7960c4c-bfe9-4693-afc2-0265709b9b8e}} , provides the accuracy (%) comparison of existing data augmentation methods with SOTAs.


Massive MIMO fingerprint-based positioning

Deep spatio-temporal residual networks (CNN-based)

Online framework for data stream processing

Exploitation of spatio-temporal characteristics of mobility events

Requires less system tuning or filtering effort

Handles unlabeled data; reinforcement learning aided semi-supervised learning

DeepSpace: An Online Deep Learning Framework for Mobile Big Data to Understand Human Mobility Patterns

Abstract: In the recent years, the rapid spread of mobile device has create the vast amount of mobile data. However, some shallow-structure models such as support vector machine (SVM) have difficulty dealing with high dimensional data with the development of mobile network. In this paper, we analyze mobile data to predict human trajectories in order to understand human mobility pattern via a deep-structure model called "DeepSpace". To the best of out knowledge, it is the first time that the deep learning approach is applied to predicting human trajectories. Furthermore, we develop the vanilla convolutional neural network (CNN) to be an online learning system, which can deal with the continuous mobile data stream. In general, "DeepSpace" consists of two different prediction models corresponding to different scales in space (the coarse prediction model and fine prediction models). This two models constitute a hierarchical structure, which enable the whole architecture to be run in parallel. Finally, we test our model based on the data usage detail records (UDRs) from the mobile cellular network in a city of southeastern China, instead of the call detail records (CDRs) which are widely used by others as usual. The experiment results show that "DeepSpace" is promising in human trajectories prediction.

Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction

Abstract: Forecasting the flow of crowds is of great importance to traffic management and public safety, yet a very challenging task affected by many complex factors, such as inter-region traffic, events and weather. In this paper, we propose a deep-learning-based approach, called ST-ResNet, to collectively forecast the in-flow and out-flow of crowds in each and every region through a city. We design an end-to-end structure of ST-ResNet based on unique properties of spatio-temporal data. More specifically, we employ the framework of the residual neural networks to model the temporal closeness, period, and trend properties of the crowd traffic, respectively. For each property, we design a branch of residual convolutional units, each of which models the spatial properties of the crowd traffic. ST-ResNet learns to dynamically aggregate the output of the three residual neural networks based on data, assigning different weights to different branches and regions. The aggregation is further combined with external factors, such as weather and day of the week, to predict the final traffic of crowds in each and every region. We evaluate ST-ResNet based on two types of crowd flows in Beijing and NYC, finding that its performance exceeds six well-know methods.

Low-effort place recognition with WiFi fingerprints using deep learning

Abstract: Using WiFi signals for indoor localization is the main localization modality of the existing personal indoor localization systems operating on mobile devices. WiFi fingerprinting is also used for mobile robots, as WiFi signals are usually available indoors and can provide rough initial position estimate or can be used together with other positioning systems. Currently, the best solutions rely on filtering, manual data analysis, and time-consuming parameter tuning to achieve reliable and accurate localization. In this work, we propose to use deep neural networks to significantly lower the work-force burden of the localization system design, while still achieving satisfactory results. Assuming the state-of-the-art hierarchical approach, we employ the DNN system for building/floor classification. We show that stacked autoencoders allow to efficiently reduce the feature space in order to achieve robust and precise classification. The proposed architecture is verified on the publicly available UJIIndoorLoc dataset and the results are compared with other solutions.

Semi-supervised Deep Reinforcement Learning in Support of IoT and Smart City Services

Abstract: Smart services are an important element of the smart cities and the Internet of Things (IoT) ecosystems where the intelligence behind the services is obtained and improved through the sensory data. Providing a large amount of training data is not always feasible; therefore, we need to consider alternative ways that incorporate unlabeled data as well. In recent years, Deep reinforcement learning (DRL) has gained great success in several application domains. It is an applicable method for IoT and smart city scenarios where auto-generated data can be partially labeled by users' feedback for training purposes. In this paper, we propose a semi-supervised deep reinforcement learning model that fits smart city applications as it consumes both labeled and unlabeled data to improve the performance and accuracy of the learning agent. The model utilizes Variational Autoencoders (VAE) as the inference engine for generalizing optimal policies. To the best of our knowledge, the proposed model is the first investigation that extends deep reinforcement learning to the semi-supervised paradigm. As a case study of smart city applications, we focus on smart buildings and apply the proposed model to the problem of indoor localization based on BLE signal strength. Indoor localization is the main component of smart city services since people spend significant time in indoor environments. Our model learns the best action policies that lead to a close estimation of the target locations with an improvement of 23% in terms of distance to the target and at least 67% more received rewards compared to the supervised DRL model.

Deep Convolutional Neural Networks for Massive MIMO Fingerprint-Based Positioning

Abstract: This paper provides an initial investigation on the application of convolutional neural networks (CNNs) for fingerprint-based positioning using measured massive MIMO channels. When represented in appropriate domains, massive MIMO channels have a sparse structure which can be efficiently learned by CNNs for positioning purposes. We evaluate the positioning accuracy of state-of-the-art CNNs with channel fingerprints generated from a channel model with a rich clustered structure: the COST 2100 channel model. We find that moderately deep CNNs can achieve fractional-wavelength positioning accuracies, provided that an enough representative data set is available for training.

A summary of work on deep learning driven mobility analysis and user localization.

Deep Learning Driven Mobility Analysis and Localization

Understanding movement patterns of groups of human beings is becoming crucial for epidemiology, urban planning, public service provisioning, and mobile network resource management {{cite:b567d97}}. Location-based services and applications (e.g. mobile AR, GPS) demand precise individual positioning technology {{cite:5f9db6b}}. As a result, research on user localization is evolving rapidly and numerous techniques are emerging {{cite:84d6fb9}}. In this subsection, we discuss research in this space, which we first summarize in Table {{table:e43c11ad-d6b1-4bee-8a2c-130e54e00006}} .


Using Deep Learning for Image-Based Plant Disease Detection

Abstract: Crop diseases are a major threat to food security, but their rapid identification remains difficult in many parts of the world due to the lack of the necessary infrastructure. The combination of increasing global smartphone penetration and recent advances in computer vision made possible by deep learning has paved the way for smartphone-assisted disease diagnosis. Using a public dataset of 54,306 images of diseased and healthy plant leaves collected under controlled conditions, we train a deep convolutional neural network to identify 14 crop species and 26 diseases (or absence thereof). The trained model achieves an accuracy of 99.35% on a held-out test set, demonstrating the feasibility of this approach. When testing the model on a set of images collected from trusted online sources - i.e. taken under conditions different from the images used for training - the model still achieves an accuracy of 31.4%. While this accuracy is much higher than the one based on random selection (2.6%), a more diverse set of training data is needed to improve the general accuracy. Overall, the approach of training deep learning models on increasingly large and publicly available image datasets presents a clear path towards smartphone-assisted crop disease diagnosis on a massive global scale.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we address the challenge of land use and land cover classification using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible provided in the Earth observation program Copernicus. We present a novel dataset based on Sentinel-2 satellite images covering 13 spectral bands and consisting out of 10 classes with in total 27,000 labeled and geo-referenced images. We provide benchmarks for this novel dataset with its spectral bands using state-of-the-art deep Convolutional Neural Network (CNNs). With the proposed novel dataset, we achieved an overall classification accuracy of 98.57%. The resulting classification system opens a gate towards a number of Earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)

Abstract: This article describes the design, implementation, and results of the latest installment of the dermoscopic image analysis benchmark challenge. The goal is to support research and development of algorithms for automated diagnosis of melanoma, the most lethal skin cancer. The challenge was divided into 3 tasks: lesion segmentation, feature detection, and disease classification. Participation involved 593 registrations, 81 pre-submissions, 46 finalized submissions (including a 4-page manuscript), and approximately 50 attendees, making this the largest standardized and comparative study in this field to date. While the official challenge duration and ranking of participants has concluded, the dataset snapshots remain available for further research and development.

Information of datasets for cross-domain few-shot benchmarks.

For the cross-domain few-shot benchmarks, we use eight different datasets. We describe the dataset information in Table {{table:408c9118-897b-4c9b-8587-0685a1c57561}} .
We use the dataset split described in {{cite:991b632}} for the benchmark of high-similarity and we use the dataset split described in {{cite:b314be5}} for the benchmark of low-similarity.
Because we do not perform the meta-training procedure using the datasets of cross-domain benchmarks, we only utilize the meta-test splits on these datasets. We use the {{formula:7a24d83e-f9f9-46b0-9ecd-530c4d3be27d}}  resized samples for evaluation on small-scale experiments.