Long-term dependent, Non-stationary,Nonlinear, Non-seasonal, Trend, Non-Gaussian

Monash Time Series Forecasting Archive

Abstract: Many businesses and industries nowadays rely on large quantities of time series data making time series forecasting an important research area. Global forecasting models that are trained across sets of time series have shown a huge potential in providing accurate forecasts compared with the traditional univariate forecasting models that work on isolated series. However, there are currently no comprehensive time series archives for forecasting that contain datasets of time series from similar sources available for the research community to evaluate the performance of new global forecasting algorithms over a wide variety of datasets. In this paper, we present such a comprehensive time series forecasting archive containing 20 publicly available time series datasets from varied domains, with different characteristics in terms of frequency, series lengths, and inclusion of missing values. We also characterise the datasets, and identify similarities and differences among them, by conducting a feature analysis. Furthermore, we present the performance of a set of standard baseline forecasting methods over all datasets across eight error metrics, for the benefit of researchers using the archive to benchmark their forecasting algorithms.

State-Level COVID-19 Symptom Searches and Case Data: Quantitative Analysis of Political Affiliation as a Predictor for Lag Time Using Google Trends and Centers for Disease Control and Prevention Data

Abstract: Background Across each state, the emergence of the COVID-19 pandemic in the United States was marked by policies and rhetoric that often corresponded to the political party in power. These diverging responses have sparked broad ongoing discussion about how the political leadership of a state may affect not only the COVID-19 case numbers in a given state but also the subjective individual experience of the pandemic. Objective This study leverages state-level data from Google Search Trends and Centers for Disease Control and Prevention (CDC) daily case data to investigate the temporal relationship between increases in relative search volume for COVID-19 symptoms and corresponding increases in case data. I aimed to identify whether there are state-level differences in patterns of lag time across each of the 4 spikes in the data (RQ1) and whether the political climate in a given state is associated with these differences (RQ2). Methods Using publicly available data from Google Trends and the CDC, linear mixed modeling was utilized to account for random state-level intercepts. Lag time was operationalized as number of days between a peak (a sustained increase before a sustained decline) in symptom search data and a corresponding spike in case data and was calculated manually for each of the 4 spikes in individual states. Google offers a data set that tracks the relative search incidence of more than 400 potential COVID-19 symptoms, which is normalized on a 0-100 scale. I used the CDC’s definition of the 11 most common COVID-19 symptoms and created a single construct variable that operationalizes symptom searches. To measure political climate, I considered the proportion of 2020 Trump popular votes in a state as well as a dummy variable for the political party that controls the governorship and a continuous variable measuring proportional party control of federal Congressional representatives. Results The strongest overall fit was for a linear mixed model that included proportion of 2020 Trump votes as the predictive variable of interest and included controls for mean daily cases and deaths as well as population. Additional political climate variables were discarded for lack of model fit. Findings indicated evidence that there are statistically significant differences in lag time by state but that no individual variable measuring political climate was a statistically significant predictor of these differences. Conclusions Given that there will likely be future pandemics within this political climate, it is important to understand how political leadership affects perceptions of and corresponding responses to public health crises. Although this study did not fully model this relationship, I believe that future research can build on the state-level differences that I identified by approaching the analysis with a different theoretical model, method for calculating lag time, or level of geographic modeling.

Global characteristics of the fourteen time series datasets with varying frequencies

Analysing Global Characteristics in the datasets

The fourteen open access datasets considered in the study are free from missing observations and are of different lengths. We have examined several global characteristics such as long-term dependency, non-stationarity, nonlinearity, seasonality, trend, and normality of these datasets using Hurst exponent, KPSS test, Teräsvirta test (nonlinearity test), Ollech and Webel's combined seasonality test, Mann-Kendall test, and Anderson-Darling test respectively to understand their structural patterns {{cite:e9514d2}}. We performed all the statistical tests using R statistical software. Table {{table:56a17fcc-aae9-4545-ab15-ccd2578a4bb9}}  summarizes the behavior of the datasets in performing the required statistical tests. All these time series datasets reflect long-range dependence, non-stationarity, and non-seasonal patterns.


various SSM, simple graph embeddings, KG embeddings and ontology embeddings

Represents ontology axioms as sentences and trains a word2vec model to generate embeddings. Evaluated on a PPI task.

Extends Onto2Vec by including informal information such as class descriptions in the axiom sentences.

Adapts translational KG embeddings for ontologies by accounting for hierarchical relations. Evaluated on a relation prediction task.

Combines concepts from OPA2Vec with biased random walks on the ontology graph.

Combines several biomedical ontologies along with annotations and applies several embeddings for the task of gene-disease prediction.

Survey on ML with ontologies. Covers both traditional SSM and recent ontology embedding methods.

Onto2Vec: joint vector-based representation of biological entities and their ontology-based annotations

Abstract: Motivation Biological knowledge is widely represented in the form of ontology‐based annotations: ontologies describe the phenomena assumed to exist within a domain, and the annotations associate a (kind of) biological entity with a set of phenomena within the domain. The structure and information contained in ontologies and their annotations make them valuable for developing machine learning, data analysis and knowledge extraction algorithms; notably, semantic similarity is widely used to identify relations between biological entities, and ontology‐based annotations are frequently used as features in machine learning applications. Results We propose the Onto2Vec method, an approach to learn feature vectors for biological entities based on their annotations to biomedical ontologies. Our method can be applied to a wide range of bioinformatics research problems such as similarity‐based prediction of interactions between proteins, classification of interaction types using supervised learning, or clustering. To evaluate Onto2Vec, we use the gene ontology (GO) and jointly produce dense vector representations of proteins, the GO classes to which they are annotated, and the axioms in GO that constrain these classes. First, we demonstrate that Onto2Vec‐generated feature vectors can significantly improve prediction of protein‐protein interactions in human and yeast. We then illustrate how Onto2Vec representations provide the means for constructing data‐driven, trainable semantic similarity measures that can be used to identify particular relations between proteins. Finally, we use an unsupervised clustering approach to identify protein families based on their Enzyme Commission numbers. Our results demonstrate that Onto2Vec can generate high quality feature vectors from biological entities and ontologies. Onto2Vec has the potential to significantly outperform the state‐of‐the‐art in several predictive applications in which ontologies are involved. Availability and implementation https://github.com/bio‐ontology‐research‐group/onto2vec

OPA2Vec: combining formal and informal content of biomedical ontologies to improve similarity-based prediction

Abstract: Motivation: Ontologies are widely used in biology for data annotation, integration, and analysis. In addition to formally structured axioms, ontologies contain meta-data in the form of annotation axioms which provide valuable pieces of information that characterize ontology classes. Annotations commonly used in ontologies include class labels, descriptions, or synonyms. Despite being a rich source of semantic information, the ontology meta-data are generally unexploited by ontology-based analysis methods such as semantic similarity measures. Results: We propose a novel method, OPA2Vec, to generate vector representations of biological entities in ontologies by combining formal ontology axioms and annotation axioms from the ontology meta-data. We apply a Word2Vec model that has been pre-trained on PubMed abstracts to produce feature vectors from our collected data. We validate our method in two different ways: first, we use the obtained vector representations of proteins as a similarity measure to predict protein-protein interaction (PPI) on two different datasets. Second, we evaluate our method on predicting gene-disease associations based on phenotype similarity by generating vector representations of genes and diseases using a phenotype ontology, and applying the obtained vectors to predict gene-disease associations. These two experiments are just an illustration of the possible applications of our method. OPA2Vec can be used to produce vector representations of any biomedical entity given any type of biomedical ontology. Availability: https://github.com/bio-ontology-research-group/opa2vec Contact: robert.hoehndorf@kaust.edu.sa and xin.gao@kaust.edu.sa.

On2Vec: Embedding-based Relation Prediction for Ontology Population

Abstract: Populating ontology graphs represents a long-standing problem for the Semantic Web community. Recent advances in translation-based graph embedding methods for populating instance-level knowledge graphs lead to promising new approaching for the ontology population problem. However, unlike instance-level graphs, the majority of relation facts in ontology graphs come with comprehensive semantic relations, which often include the properties of transitivity and symmetry, as well as hierarchical relations. These comprehensive relations are often too complex for existing graph embedding methods, and direct application of such methods is not feasible. Hence, we propose On2Vec, a novel translation-based graph embedding method for ontology population. On2Vec integrates two model components that effectively characterize comprehensive relation facts in ontology graphs. The first is the Component-specific Model that encodes concepts and relations into low-dimensional embedding spaces without a loss of relational properties; the second is the Hierarchy Model that performs focused learning of hierarchical relation facts. Experiments on several well-known ontology graphs demonstrate the promising capabilities of On2Vec in predicting and verifying new relation facts. These promising results also make possible significant improvements in related methods.

OWL2Vec*: Embedding of OWL Ontologies

Abstract: Semantic embedding of knowledge graphs has been widely studied and used for prediction and statistical analysis tasks across various domains such as Natural Language Processing and the Semantic Web. However, less attention has been paid to developing robust methods for embedding OWL (Web Ontology Language) ontologies which can express a much wider range of semantics than knowledge graphs and have been widely adopted in domains such as bioinformatics. In this paper, we propose a random walk and word embedding based ontology embedding method named OWL2Vec*, which encodes the semantics of an OWL ontology by taking into account its graph structure, lexical information and logical constructors. Our empirical evaluation with three real world datasets suggests that OWL2Vec* benefits from these three different aspects of an ontology in class membership prediction and class subsumption prediction tasks. Furthermore, OWL2Vec* often significantly outperforms the state-of-the-art methods in our experiments.

Predicting Gene-Disease Associations with Knowledge Graph Embeddings over Multiple Ontologies

Abstract: Ontology-based approaches for predicting gene-disease associations include the more classical semantic similarity methods and more recently knowledge graph embeddings. While semantic similarity is typically restricted to hierarchical relations within the ontology, knowledge graph embeddings consider their full breadth. However, embeddings are produced over a single graph and complex tasks such as gene-disease association may require additional ontologies. We investigate the impact of employing richer semantic representations that are based on more than one ontology, able to represent both genes and diseases and consider multiple kinds of relations within the ontologies. Our experiments demonstrate the value of employing knowledge graph embeddings based on random-walks and highlight the need for a closer integration of different ontologies.

Semantic similarity and machine learning with ontologies

Abstract: Abstract Ontologies have long been employed in the life sciences to formally represent and reason over domain knowledge and they are employed in almost every major biological database. Recently, ontologies are increasingly being used to provide background knowledge in similarity-based analysis and machine learning models. The methods employed to combine ontologies and machine learning are still novel and actively being developed. We provide an overview over the methods that use ontologies to compute similarity and incorporate them in machine learning methods; in particular, we outline how semantic similarity measures and ontology embeddings can exploit the background knowledge in ontologies and how ontologies can provide constraints that improve machine learning models. The methods and experiments we describe are available as a set of executable notebooks, and we also provide a set of slides and additional resources at https://github.com/bio-ontology-research-group/machine-learning-with-ontologies.

Overview of some of the related approaches, their aim and a short description. The horizontal lines separate different types of research articles.

Mining ontologies with network-based approaches

A more extensive overview of related work is summarized in Table {{table:1b82d5b5-d700-41d9-b2ea-0c2433fdde74}} .


A dense road disparity image is transformed to better distinguish the damaged and undamaged road areas. The transformed disparity image is segmented using Otus's thresholding method for road pothole detection.

SLIC is utilized to group the transformed disparities into a collection of superpixels. The road potholes are then detected by finding the superpixels, whose values are lower than an adaptively determined threshold.

Road Damage Detection Based on Unsupervised Disparity Map Segmentation

Abstract: This article presents a novel road damage detection algorithm based on unsupervised disparity map segmentation. Firstly, a disparity map is transformed by minimizing an energy function with respect to stereo rig roll angle and road disparity projection model. Instead of solving this energy minimization problem using non-linear optimization techniques, we directly find its numerical solution. The transformed disparity map is then segmented using Otus’s thresholding method, and the damaged road areas can be extracted. The proposed algorithm requires no parameters when detecting road damage. The experimental results illustrate that our proposed algorithm performs both accurately and efficiently. The pixel-level road damage detection accuracy is approximately 97.56%. The source code is publicly available at: https://github.com/ruirangerfan/unsupervised_disparity_map_segmentation.git.

Rethinking Road Surface 3D Reconstruction and Pothole Detection: From Perspective Transformation to Disparity Map Segmentation

Abstract: Potholes are one of the most common forms of road damage, which can severely affect driving comfort, road safety and vehicle condition. Pothole detection is typically performed by either structural engineers or certified inspectors. This task is, however, not only hazardous for the personnel but also extremely time-consuming. This paper presents an efficient pothole detection algorithm based on road disparity map estimation and segmentation. We first generalize the perspective transformation by incorporating the stereo rig roll angle. The road disparities are then estimated using semi-global matching. A disparity map transformation algorithm is then performed to better distinguish the damaged road areas. Finally, we utilize simple linear iterative clustering to group the transformed disparities into a collection of superpixels. The potholes are then detected by finding the superpixels, whose values are lower than an adaptively determined threshold. The proposed algorithm is implemented on an NVIDIA RTX 2080 Ti GPU in CUDA. The experiments demonstrate the accuracy and efficiency of our proposed road pothole detection algorithm, where an accuracy of 99.6% and an F-score of 89.4% are achieved.

Representative classical 2-D image processing-based approaches.

Classical 2-D image processing-based road pothole detection is a well-researched topic. As shown in Fig. {{figure:b7f2df65-3d11-4f2c-8fd9-25997a87b9ee}} , such approaches generally have a four-stage pipeline: (1) image pre-processing, (2) image segmentation, (3) damaged area extraction, and (4) detection result post-processing {{cite:43c0069}}. The representative prior arts are summarized in Table {{table:4ae28882-8cd0-49d5-be65-a78417b2bc59}} .


HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish

Abstract: BERT-based models are currently used for solving nearly all Natural Language Processing (NLP) tasks and most often achieve state-of-the-art results. Therefore, the NLP community conducts extensive research on understanding these models, but above all on designing effective and efficient training procedures. Several ablation studies investigating how to train BERT-like models have been carried out, but the vast majority of them concerned only the English language. A training procedure designed for English does not have to be universal and applicable to other especially typologically different languages. Therefore, this paper presents the first ablation study focused on Polish, which, unlike the isolating English language, is a fusional language. We design and thoroughly evaluate a pretraining procedure of transferring knowledge from multilingual to monolingual BERT-based models. In addition to multilingual model initialization, other factors that possibly influence pretraining are also explored, i.e. training objective, corpus size, BPE-Dropout, and pretraining length. Based on the proposed procedure, a Polish BERT-based language model – HerBERT – is trained. This model achieves state-of-the-art results on multiple downstream tasks.

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Abstract: BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences are fed into the network, which causes a massive computational overhead: Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT. The construction of BERT makes it unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. In this publication, we present Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT. We evaluate SBERT and SRoBERTa on common STS tasks and transfer learning tasks, where it outperforms other state-of-the-art sentence embeddings methods.

Language Models used for experiments. All models can be accessed via the HuggingFace repository.

The next step was to select initial models in the benchmark to provide a baseline and allow for easy comparison with the already published models. Each baseline model had to be available in the HuggingFace repository, and it could not be too big since we were again limited by the amount of available compute.Maximum size of the model to perform hyperparameter search in a reasonable time was 350 million parameters
We first took the models of different sizes provided in KLEJ – HerBERT-large is a top-1 model in the KLEJ benchmark. We used these models and our hyperparameter search module to validate the experimental setup and generate the first results for benchmark. We then utilized another popular mono-lingual encoder model and tested it against the previous one. Finally, to provide some diversity, we took the sentence model (which in fact is a cross-lingual) and ran the experiment. We plan to add new models to the benchmark to allow comparison of cross- vs. single-lingual and sentence- vs. word encoders. Table {{table:38c574af-20ef-4a57-8640-2d1450cbe0b6}}  shows the final collection of models used for experiments.

We picked four recent transformer-based language models for Polish publicly available in the HuggingFace hub, along with one multilingual XLM-RoBERTa model. We present those models with their total number of parameters and repository location in Table {{table:38c574af-20ef-4a57-8640-2d1450cbe0b6}} . For fine-tuning, we utilize sequence and token classification models from the transformers library {{cite:50f4aa2}}, consisting of a single linear classification layer with dropout. For the initial datasets evaluation, we chose both cased and uncased versions of the language models.


Attention {{cite:29046d7}}, GNN {{cite:79b4be5}}

Random Forest, DNN, GCN {{cite:ef11bb3}}, CMPNN {{cite:88941be}}

DrugBank {{cite:447910b}}, HetioNet {{cite:d184f3a}}, TWOSIDES {{cite:91fab26}}

SumGNN: Multi-typed Drug Interaction Prediction via Efficient Knowledge Graph Summarization

Abstract: Thanks to the increasing availability of drug-drug interactions (DDI) datasets and large biomedical knowledge graphs (KGs), accurate detection of adverse DDI using machine learning models becomes possible. However, it remains largely an open problem how to effectively utilize large and noisy biomedical KG for DDI detection. Due to its sheer size and amount of noise in KGs, it is often less beneficial to directly integrate KGs with other smaller but higher quality data (e.g., experimental data). Most of the existing approaches ignore KGs altogether. Some try to directly integrate KGs with other data via graph neural networks with limited success. Furthermore, most previous works focus on binary DDI prediction whereas the multi-typed DDI pharmacological effect prediction is a more meaningful but harder task. To fill the gaps, we propose a new method SumGNN: knowledge summarization graph neural network, which is enabled by a subgraph extraction module that can efficiently anchor on relevant subgraphs from a KG, a self-attention based subgraph summarization scheme to generate a reasoning path within the subgraph, and a multi-channel knowledge and data integration module that utilizes massive external biomedical knowledge for significantly improved multi-typed DDI predictions. SumGNN outperforms the best baseline by up to 5.54\%, and the performance gain is particularly significant in low data relation types. In addition, SumGNN provides interpretable prediction via the generated reasoning paths for each prediction.

Prediction of Compound Synthesis Accessibility Based on Reaction Knowledge Graph

Abstract: With the increasing application of deep-learning-based generative models for de novo molecule design, the quantitative estimation of molecular synthetic accessibility (SA) has become a crucial factor for prioritizing the structures generated from generative models. It is also useful for helping in the prioritization of hit/lead compounds and guiding retrosynthesis analysis. In this study, based on the USPTO and Pistachio reaction datasets, a chemical reaction network was constructed for the identification of the shortest reaction paths (SRP) needed to synthesize compounds, and different SRP cut-offs were then used as the threshold to distinguish a organic compound as either an easy-to-synthesize (ES) or hard-to-synthesize (HS) class. Two synthesis accessibility models (DNN-ECFP model and graph-based CMPNN model) were built using deep learning/machine learning algorithms. Compared to other existing synthesis accessibility scoring schemes, such as SYBA, SCScore, and SAScore, our results show that CMPNN (ROC AUC: 0.791) performs better than SYBA (ROC AUC: 0.76), albeit marginally, and outperforms SAScore and SCScore. Our prediction models based on historical reaction knowledge could be a potential tool for estimating molecule SA.

An overview of the recent literature utilizing KG's in healthcare applications, Drug-DrugInteraction (DDI), Drug-Target Interaction (DTI), Adverse Drug Reaction (ADR), Drug-Repurposing (DR), Drug-Design (DD). BCE: Binary Cross Entropy, CE: Cross Entropy, PR: Pointwise Ranking, FCS: Frequent Consecutive Subsequence, PCA: Principal Component Analysis, CCE: Categorical Cross Entropy, DNN: Deep Neural Network, MLP: Multiple Layer Perceptron

Table {{table:d93b7ed4-923f-409c-892e-6aa04ae7e5ec}}  illustrates the extensive utilization of KGs across diverse domains within drug research, showcasing their versatility and effectiveness.


Multilingual Classification, Sentiment Analysis

Object Recognition, Image Classification

Heterogeneous Domain Adaptation via Soft Transfer Network

Abstract: Heterogeneous domain adaptation (HDA) aims to facilitate the learning task in a target domain by borrowing knowledge from a heterogeneous source domain. In this paper, we propose a Soft Transfer Network (STN), which jointly learns a domain-shared classifier and a domain-invariant subspace in an end-to-end manner, for addressing the HDA problem. The proposed STN not only aligns the discriminative directions of domains but also matches both the marginal and conditional distributions across domains. To circumvent negative transfer, STN aligns the conditional distributions by using the soft-label strategy of unlabeled target data, which prevents the hard assignment of each unlabeled target data to only one category that may be incorrect. Further, STN introduces an adaptive coefficient to gradually increase the importance of the soft-labels since they will become more and more accurate as the number of iterations increases. We perform experiments on the transfer tasks of image-to-image, text-to-image, and text-to-text. Experimental results testify that the STN significantly outperforms several state-of-the-art approaches.

Cross-Domain Structure Preserving Projection for Heterogeneous Domain Adaptation

Abstract: Heterogeneous Domain Adaptation (HDA) addresses the transfer learning problems where data from the source and target domains are of different modalities (e.g., texts and images) or feature dimensions (e.g., features extracted with different methods). It is useful for multi-modal data analysis. Traditional domain adaptation algorithms assume that the representations of source and target samples reside in the same feature space, hence are likely to fail in solving the heterogeneous domain adaptation problem. Contemporary state-of-the-art HDA approaches are usually composed of complex optimization objectives for favourable performance and are therefore computationally expensive and less generalizable. To address these issues, we propose a novel Cross-Domain Structure Preserving Projection (CDSPP) algorithm for HDA. As an extension of the classic LPP to heterogeneous domains, CDSPP aims to learn domain-specific projections to map sample features from source and target domains into a common subspace such that the class consistency is preserved and data distributions are sufficiently aligned. CDSPP is simple and has deterministic solutions by solving a generalized eigenvalue problem. It is naturally suitable for supervised HDA but has also been extended for semi-supervised HDA where the unlabelled target domain samples are available. Extensive experiments have been conducted on commonly used benchmark datasets (i.e. Office-Caltech, Multilingual Reuters Collection, NUS-WIDE-ImageNet) for HDA as well as the Office-Home dataset firstly introduced for HDA by ourselves due to its significantly larger number of classes than the existing ones (65 vs 10, 6 and 8). The experimental results of both supervised and semi-supervised HDA demonstrate the superior performance of our proposed method against contemporary state-of-the-art methods.

The summary of datasets.

In this section, we will delve into the utilization of HTL methods in specific areas, including NLP, CV, Multimodality, and Biomedicine, as outlined in Table {{table:d116b437-bf3b-4046-924a-594c8fee7b99}}  and illustrated in Figure {{figure:a8631c52-7ef5-4404-a52d-c3dc891ee86f}} . Through a detailed examination of HTL methods in these domains, we aim to uncover the challenges and progress across diverse application contexts. Additionally, we highlight prominent datasets in HTL, providing comprehensive details and referencing the methods that employed them in Table {{table:5a33f184-c3d6-4bc9-accc-82f2488a5059}} .


Deep Image Matting

Abstract: Image matting is a fundamental computer vision problem and has many applications. Previous algorithms have poor performance when an image has similar foreground and background colors or complicated textures. The main reasons are prior methods 1) only use low-level features and 2) lack high-level context. In this paper, we propose a novel deep learning based algorithm that can tackle both these problems. Our deep model has two parts. The first part is a deep convolutional encoder-decoder network that takes an image and the corresponding trimap as inputs and predict the alpha matte of the image. The second part is a small convolutional network that refines the alpha matte predictions of the first network to have more accurate alpha values and sharper edges. In addition, we also create a large-scale image matting dataset including 49300 training images and 1000 testing images. We evaluate our algorithm on the image matting benchmark, our testing set, and a wide variety of real images. Experimental results clearly demonstrate the superiority of our algorithm over previous methods.

Deep Image Matting

Deep Image Matting

Deep Image Matting

Real-Time High-Resolution Background Matting

Abstract: We introduce a real-time, high-resolution background replacement technique which operates at 30fps in 4K resolution, and 60fps for HD on a modern GPU. Our technique is based on background matting, where an additional frame of the background is captured and used in recovering the alpha matte and the foreground layer. The main challenge is to compute a high-quality alpha matte, preserving strand-level hair details, while processing high-resolution images in real-time. To achieve this goal, we employ two neural networks; a base network computes a low-resolution result which is refined by a second network operating at high-resolution on selective patches. We introduce two large-scale video and image matting datasets: VideoMatte240K and PhotoMatte13K/85. Our approach yields higher quality results compared to the previous state-of-the-art in background matting, while simultaneously yielding a dramatic boost in both speed and resolution.

Comparison of different methods on different datasets according to our approach (POMAGAN). Please note that we tested each model with the same data (the combination of the same foreground and background) to perform a fair comparison.

Quantitative evaluation Experimental results are shown in Table {{table:df19b3e8-0b3b-4ce2-9463-5016f79ae645}} . We presented MSE, MAE, SAD, Gradient, and Connectivity, respectively, and we evaluated all models under the same conditions, e.g., used background image, resolution. According to the experimental results presented in Table {{table:df19b3e8-0b3b-4ce2-9463-5016f79ae645}} , our model surpassed the performance of MODNet, which does not use any additional input, on three different benchmarks. Other methods in the table —BGM-V2 and FBA— benefit from additional input such as the same background image without subject and trimap. These additional inputs make the task easier and more accurate results are likely to happen, since the background image is exactly the same one as the original input image, and trimap identifies most of the area on the image as foreground and background. On the AIM dataset, our model is found superior to the BGM-V2 in all metrics. However, the FBA method achieves the best performance on this test set. On the PM85 dataset, our proposed POMAGAN outperforms all methods on the MSE metric. However, BGM-V2 is slightly better in other metrics. On the D646 benchmark, we again outperform the MODNet and BGM-V2. The FBA reaches the best performance. However, it is slightly better than our method and our results are quite acceptable when we compare with the FBA. Please note that while our approach does not take any input in addition to the original image, the FBA method takes trimap and the BGM-V2 method takes the same background image with the original image without the subject itself. However, they are too sensitive against these additional inputs. For instance, if there are any dissimilarities on the background image such as translation, BGM-V2 cannot produce a proper output, generates a completely corrupted prediction instead. Similarly, since the FBA method is quite sensitive to the input trimap, different trimaps cause considerably different outputs. In addition to all these states, our model and all other models are sensitive to the background of the input image according to our detailed experiments. It indicates that the alpha map prediction performance of the models for the same subject can considerably change according to the background of the input image.


Minimize energy consumption of UDs and UAVs

Heuristic Algorithms for Co-scheduling of Edge Analytics and Routes for UAV Fleet Missions

Abstract: Unmanned Aerial Vehicles (UAVs) or drones are increasingly used for urban applications like traffic monitoring and construction surveys. Autonomous navigation allows drones to visit waypoints and accomplish activities as part of their mission. A common activity is to hover and observe a location using on-board cameras. Advances in Deep Neural Networks (DNNs) allow such videos to be analyzed for automated decision making. UAVs also host edge computing capability for on-board inferencing by such DNNs. To this end, for a fleet of drones, we propose a novel Mission Scheduling Problem (MSP) that co-schedules the flight routes to visit and record video at waypoints, and their subsequent on-board edge analytics. The proposed schedule maximizes the utility from the activities while meeting activity deadlines as well as energy and computing constraints. We first prove that MSP is NP-hard and then optimally solve it by formulating a mixed integer linear programming (MILP) problem. Next, we design two efficient heuristic algorithms, JSC and VRC, that provide fast sub-optimal solutions. Evaluation of these three schedulers using real drone traces demonstrate utility-runtime trade-offs under diverse workloads.

Mobile Edge Computing via a UAV-Mounted Cloudlet: Optimization of Bit Allocation and Path Planning

Abstract: Unmanned aerial vehicles (UAVs) have been recently considered as means to provide enhanced coverage or relaying services to mobile users (MUs) in wireless systems with limited or no infrastructure. In this paper, a UAV-based mobile cloud computing system is studied in which a moving UAV is endowed with computing capabilities to offer computation offloading opportunities to MUs with limited local processing capabilities. The system aims at minimizing the total mobile energy consumption while satisfying quality of service requirements of the offloaded mobile application. Offloading is enabled by uplink and downlink communications between the mobile devices and the UAV, which take place by means of frequency division duplex via orthogonal or nonorthogonal multiple access schemes. The problem of jointly optimizing the bit allocation for uplink and downlink communications as well as for computing at the UAV, along with the cloudlet's trajectory under latency and UAV's energy budget constraints is formulated and addressed by leveraging successive convex approximation strategies. Numerical results demonstrate the significant energy savings that can be accrued by means of the proposed joint optimization of bit allocation and cloudlet's trajectory as compared to local mobile execution as well as to partial optimization approaches that design only the bit allocation or the cloudlet's trajectory.

Energy-Efficient UAV-Assisted Mobile Edge Computing: Resource Allocation and Trajectory Optimization

Abstract: In this paper, we study unmanned aerial vehicle (UAV) assisted mobile edge computing (MEC) with the objective to optimize computation offloading with minimum UAV energy consumption. In the considered scenario, a UAV plays the role of an aerial cloudlet to collect and process the computation tasks offloaded by ground users. Given the service requirements of users, we aim to maximize UAV energy efficiency by jointly optimizing the UAV trajectory, the user transmit power, and computation load allocation. The resulting optimization problem corresponds to nonconvex fractional programming, and the Dinkelbach algorithm and the successive convex approximation (SCA) technique are adopted to solve it. Furthermore, we decompose the problem into multiple subproblems for distributed and parallel problem solving. To cope with the case when the knowledge of user mobility is limited, we adopt a spatial distribution estimation technique to predict the location of ground users so that the proposed approach can still be applied. Simulation results demonstrate the effectiveness of the proposed approach for maximizing the energy efficiency of UAV.

Towards Big data processing in IoT: Path Planning and Resource Management of UAV Base Stations in Mobile-Edge Computing System

Abstract: Heavy data load and wide cover range have always been crucial problems for online data processing in internet of things (IoT). Recently, mobile-edge computing (MEC) and unmanned aerial vehicle base stations (UAV-BSs) have emerged as promising techniques in IoT. In this paper, we propose a three-layer online data processing network based on MEC technique. On the bottom layer, raw data are generated by widely distributed sensors, which reflects local information. Upon them, unmanned aerial vehicle base stations (UAV-BSs) are deployed as moving MEC servers, which collect data and conduct initial steps of data processing. On top of them, a center cloud receives processed results and conducts further evaluation. As this is an online data processing system, the edge nodes should stabilize delay to ensure data freshness. Furthermore, limited onboard energy poses constraints to edge processing capability. To smartly manage network resources for saving energy and stabilizing delay, we develop an online determination policy based on Lyapunov Optimization. In cases of low data rate, it tends to reduce edge processor frequency for saving energy. In the presence of high data rate, it will smartly allocate bandwidth for edge data offloading. Meanwhile, hovering UAV-BSs bring a large and flexible service coverage, which results in the problem of effective path planning. In this paper, we apply deep reinforcement learning and develop an online path planning algorithm. Taking observations of around environment as input, a CNN network is trained to predict the reward of each action. By simulations, we validate its effectiveness in enhancing service coverage. The result will contribute to big data processing in future IoT.

Task and Bandwidth Allocation for UAV-Assisted Mobile Edge Computing with Trajectory Design

Abstract: In this paper, we investigate a mobile edge computing (MEC) architecture with the assistance of an unmanned aerial vehicle (UAV). The UAV acts as a computing server to help the user equipment (UEs) compute their tasks as well as a relay to further offload the UEs' tasks to the access point (AP) for computing. The total energy consumption of the UAV and UEs is minimized by jointly optimizing the task allocation, the bandwidth allocation and the UAV's trajectory, subject to the task constraints, the information-causality constraints, the bandwidth allocation constraints, and the UAV's trajectory constraints. The formulated optimization problem is nonconvex, and we propose an alternating algorithm to optimize the parameters iteratively. The effectiveness of the algorithm is verified by the simulation results, where great performance gain is achieved in comparison with some practical baselines, especially in handling the computation- intensive and latency-critical tasks.

A comparison of Joint Resource Management Studies in UEC

black
In this section, we discuss the key performance indicators, i.e., energy consumption, throughput, latency, resource utilization, utility and cost in UEC for resource management. These indicators are identified based on the objectives presented in table {{table:9579846a-ffde-4031-abfe-c8222b5d18a3}}  - {{table:37dfd062-16b8-4497-b250-8d4611efe97b}} .


Handcrafted HOG+HOF {{cite:cbeaebf}} and automatic CNN extracted features

Automatically extracted optical flow, intensity and gradient features.

Reconstruction error between predicted and actual output.

Anomaly score, namely regularity score derived using reconstruction error in autoencoders.

Anomaly detection with a combination of semantic features using (a)Nearest neighbor-based method (NN), (b)OCSVM and (c) KDE.

Anomaly score as a fusion of Optical-flow and appearance reconstruction error.

Contextual information embedded in trajectory thermal transfer fields using OCSVM.

Peak Signal to Noise Ratio (PSNR) score based on optical flow, intensity, gradient loss.

Abnormality score derived using the losses of the generator and the discriminator.

An anomaly score using sparsity and smoothness constraints.

Effective for recognizing abnormalities when the training data is loosely supervised to contain mostly normal events.

A regularity score, used as a measure of normalcy in a scene, derived using both hand crafted features and automatic features using fully convolutional feed-forward
autoencoder.

It addresses the problem of joint detection and
recounting of abnormal events in videos in presence of false alarms.

Global and Local anomaly detection in crowded scene.

This is first kind of anomaly detection done using thermal fields that can detect contextual anomalies.

DNN-based prediction ({{cite:8a88596}}) and GAN {{cite:bd85df7}} based discriminator applied on optical flow frames derived using ({{cite:a89c21c}}) to detect robustness to the uncertainty in normal events and the sensitivity to abnormal events.

Can detect anomalies from dataset containing complex motion and frequent occlusions.

A generic method applied on a variety of real-life scenarios.

Anomaly Detection in Video Using Predictive Convolutional Long Short-Term Memory Networks

Abstract: Automating the detection of anomalous events within long video sequences is challenging due to the ambiguity of how such events are defined. We approach the problem by learning generative models that can identify anomalies in videos using limited supervision. We propose end-to-end trainable composite Convolutional Long Short-Term Memory (Conv-LSTM) networks that are able to predict the evolution of a video sequence from a small number of input frames. Regularity scores are derived from the reconstruction errors of a set of predictions with abnormal video sequences yielding lower regularity scores as they diverge further from the actual sequence over time. The models utilize a composite structure and examine the effects of conditioning in learning more meaningful representations. The best model is chosen based on the reconstruction and prediction accuracy. The Conv-LSTM models are evaluated both qualitatively and quantitatively, demonstrating competitive results on anomaly detection datasets. Conv-LSTM units are shown to be an effective tool for modeling and predicting video sequences.

Learning Temporal Regularity in Video Sequences

Abstract: Perceiving meaningful activities in a long video sequence is a challenging problem due to ambiguous definition of 'meaningfulness' as well as clutters in the scene. We approach this problem by learning a generative model for regular motion patterns (termed as regularity) using multiple sources with very limited supervision. Specifically, we propose two methods that are built upon the autoencoders for their ability to work with little to no supervision. We first leverage the conventional handcrafted spatio-temporal local features and learn a fully connected autoencoder on them. Second, we build a fully convolutional feed-forward autoencoder to learn both the local features and the classifiers as an end-to-end learning framework. Our model can capture the regularities from multiple datasets. We evaluate our methods in both qualitative and quantitative ways - showing the learned regularity of videos in various aspects and demonstrating competitive performance on anomaly detection datasets as an application.

Joint Detection and Recounting of Abnormal Events by Learning Deep Generic Knowledge

Abstract: This paper addresses the problem of joint detection and recounting of abnormal events in videos. Recounting of abnormal events, i.e., explaining why they are judged to be abnormal, is an unexplored but critical task in video surveillance, because it helps human observers quickly judge if they are false alarms or not. To describe the events in the human-understandable form for event recounting, learning generic knowledge about visual concepts (e.g., object and action) is crucial. Although convolutional neural networks (CNNs) have achieved promising results in learning such concepts, it remains an open question as to how to effectively use CNNs for abnormal event detection, mainly due to the environment-dependent nature of the anomaly detection. In this paper, we tackle this problem by integrating a generic CNN model and environment-dependent anomaly detectors. Our approach first learns CNN with multiple visual tasks to exploit semantic information that is useful for detecting and recounting abnormal events. By appropriately plugging the model into anomaly detectors, we can detect and recount abnormal events while taking advantage of the discriminative power of CNNs. Our approach outperforms the state-of-the-art on Avenue and UCSD Ped2 benchmarks for abnormal event detection and also produces promising results of abnormal event recounting.

Abnormal event detection in videos using generative adversarial nets

Abstract: In this paper we address the abnormality detection problem in crowded scenes. We propose to use Generative Adversarial Nets (GANs), which are trained using normal frames and corresponding optical-flow images in order to learn an internal representation of the scene normality. Since our GANs are trained with only normal data, they are not able to generate abnormal events. At testing time the real data are compared with both the appearance and the motion representations reconstructed by our GANs and abnormal areas are detected by computing local differences. Experimental results on challenging abnormality detection datasets show the superiority of the proposed method compared to the state of the art in both frame-level and pixel-level abnormality detection tasks.

A Tube-and-Droplet-based Approach for Representing and Analyzing Motion Trajectories

Abstract: Trajectory analysis is essential in many applications. In this paper, we address the problem of representing motion trajectories in a highly informative way, and consequently utilize it for analyzing trajectories. Our approach first leverages the complete information from given trajectories to construct a thermal transfer field which provides a context-rich way to describe the global motion pattern in a scene. Then, a 3D tube is derived which depicts an input trajectory by integrating its surrounding motion patterns contained in the thermal transfer field. The 3D tube effectively: 1) maintains the movement information of a trajectory, 2) embeds the complete contextual motion pattern around a trajectory, 3) visualizes information about a trajectory in a clear and unified way. We further introduce a droplet-based process. It derives a droplet vector from a 3D tube, so as to characterize the high-dimensional 3D tube information in a simple but effective way. Finally, we apply our tube-and-droplet representation to trajectory analysis applications including trajectory clustering, trajectory classification & abnormality detection, and 3D action recognition. Experimental comparisons with state-of-the-art algorithms demonstrate the effectiveness of our approach.

Future Frame Prediction for Anomaly Detection - A New Baseline

Abstract: Anomaly detection in videos refers to the identification of events that do not conform to expected behavior. However, almost all existing methods tackle the problem by minimizing the reconstruction errors of training data, which cannot guarantee a larger reconstruction error for an abnormal event. In this paper, we propose to tackle the anomaly detection problem within a video prediction framework. To the best of our knowledge, this is the first work that leverages the difference between a predicted future frame and its ground truth to detect an abnormal event. To predict a future frame with higher quality for normal events, other than the commonly used appearance (spatial) constraints on intensity and gradient, we also introduce a motion (temporal) constraint in video prediction by enforcing the optical flow between predicted frames and ground truth frames to be consistent, and this is the first work that introduces a temporal constraint into the video prediction task. Such spatial and motion constraints facilitate the future frame prediction for normal events, and consequently facilitate to identify those abnormal events that do not conform the expectation. Extensive experiments on both a toy dataset and some publicly available datasets validate the effectiveness of our method in terms of robustness to the uncertainty in normal events and the sensitivity to abnormal events. All codes are released in https://github.com/StevenLiuWen/ano_pred_cvpr2018.

STAN: Spatio-Temporal Adversarial Networks for Abnormal Event Detection

Abstract: In this paper, we propose a novel abnormal event detection method with spatio-temporal adversarial networks (STAN). We devise a spatio-temporal generator which synthesizes an inter-frame by considering spatio-temporal characteristics with bidirectional ConvLSTM. A proposed spatio-temporal discriminator determines whether an input sequence is real-normal or not with 3D convolutional layers. These two networks are trained in an adversarial way to effectively encode spatio-temporal features of normal patterns. After the learning, the generator and the discriminator can be independently used as detectors, and deviations from the learned normal patterns are detected as abnormalities. Experimental results show that the proposed method achieved competitive performance compared to the state-of-the-art methods. Further, for the interpretation, we visualize the location of abnormal events detected by the proposed networks using a generator loss and discriminator gradients.

Real-world Anomaly Detection in Surveillance Videos

Abstract: Surveillance videos are able to capture a variety of realistic anomalies. In this paper, we propose to learn anomalies by exploiting both normal and anomalous videos. To avoid annotating the anomalous segments or clips in training videos, which is very time consuming, we propose to learn anomaly through the deep multiple instance ranking framework by leveraging weakly labeled training videos, i.e. the training labels (anomalous or normal) are at video-level instead of clip-level. In our approach, we consider normal and anomalous videos as bags and video segments as instances in multiple instance learning (MIL), and automatically learn a deep anomaly ranking model that predicts high anomaly scores for anomalous video segments. Furthermore, we introduce sparsity and temporal smoothness constraints in the ranking loss function to better localize anomaly during training. We also introduce a new large-scale first of its kind dataset of 128 hours of videos. It consists of 1900 long and untrimmed real-world surveillance videos, with 13 realistic anomalies such as fighting, road accident, burglary, robbery, etc. as well as normal activities. This dataset can be used for two tasks. First, general anomaly detection considering all anomalies in one group and all normal activities in another group. Second, for recognizing each of 13 anomalous activities. Our experimental results show that our MIL method for anomaly detection achieves significant improvement on anomaly detection performance as compared to the state-of-the-art approaches. We provide the results of several recent deep learning baselines on anomalous activity recognition. The low recognition performance of these baselines reveals that our dataset is very challenging and opens more opportunities for future work. The dataset is available at: https://webpages.uncc.edu/cchen62/dataset.html

Representative work based on used features

Broadly, the features can be classified as object oriented and non-object oriented. The classification is represented in Fig. {{figure:3402606e-4bab-4036-91a2-7cf387a6fecd}} . Using object oriented features, anomalies can be detected by extracting the objects {{cite:096a100}}, {{cite:0a67850}} or trajectories {{cite:c88ca27}}, {{cite:9318646}}, {{cite:b4902a0}}. Objects or trajectories represented in the form of feature descriptors become the data for anomaly detection. In the latter approach, low-level descriptors for pixel or pixel group features, intensities, optical flows, or resultant features from spatio-temporal cubes (STC) {{cite:4bf665d}}, {{cite:fafa270}}, {{cite:60845e1}}, {{cite:1c4c956}}, {{cite:45a2263}}, {{cite:9f69df8}}, {{cite:d0a9b66}} have been used for anomaly detection. Some methods use hybrid features for anomaly detections {{cite:c9a9db8}}, {{cite:dbf4d06}}, {{cite:271fb35}}, {{cite:c3b2afb}}. Some of the important work using various aforementioned features are summarized in Table {{table:784760eb-6081-4b4a-b6bf-8e9f6496dbaf}} .


Argoverse: 3D Tracking and Forecasting With Rich Maps

Abstract: We present Argoverse, a dataset designed to support autonomous vehicle perception tasks including 3D tracking and motion forecasting. Argoverse includes sensor data collected by a fleet of autonomous vehicles in Pittsburgh and Miami as well as 3D tracking annotations, 300k extracted interesting vehicle trajectories, and rich semantic maps. The sensor data consists of 360 degree images from 7 cameras with overlapping fields of view, forward-facing stereo imagery, 3D point clouds from long range LiDAR, and 6-DOF pose. Our 290km of mapped lanes contain rich geometric and semantic metadata which are not currently available in any public dataset. All data is released under a Creative Commons license at Argoverse.org. In baseline experiments, we use map information such as lane direction, driveable area, and ground height to improve the accuracy of 3D object tracking. We use 3D object tracking to mine for more than 300k interesting vehicle trajectories to create a trajectory forecasting benchmark. Motion forecasting experiments ranging in complexity from classical methods (k-NN) to LSTMs demonstrate that using detailed vector maps with lane-level information substantially reduces prediction error. Our tracking and forecasting experiments represent only a superficial exploration of the potential of rich maps in robotic perception. We hope that Argoverse will enable the research community to explore these problems in greater depth.

nuScenes: A Multimodal Dataset for Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.

Scalability in Perception for Autonomous Driving: Waymo Open Dataset

Abstract: The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the over-all viability of the technology. In an effort to help align the research community’s contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.

KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D

Abstract: For the last few decades, several major subfields of artificial intelligence including computer vision, graphics, and robotics have progressed largely independently from each other. Recently, however, the community has realized that progress towards robust intelligent systems such as self-driving cars requires a concerted effort across the different fields. This motivated us to develop KITTI-360, successor of the popular KITTI dataset. KITTI-360 is a suburban driving dataset which comprises richer input modalities, comprehensive semantic instance annotations and accurate localization to facilitate research at the intersection of vision, graphics and robotics. For efficient annotation, we created a tool to label 3D scenes with bounding primitives and developed a model that transfers this information into the 2D image domain, resulting in over 150k images and 1B 3D points with coherent semantic instance annotations across 2D and 3D. Moreover, we established benchmarks and baselines for several tasks relevant to mobile perception, encompassing problems from computer vision, graphics, and robotics on the same dataset, e.g., semantic scene understanding, novel view synthesis and semantic SLAM. KITTI-360 will enable progress at the intersection of these research areas and thus contribute towards solving one of today's grand challenges: the development of fully autonomous self-driving systems.

Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting

Abstract: We introduce Argoverse 2 (AV2) - a collection of three datasets for perception and forecasting research in the self-driving domain. The annotated Sensor Dataset contains 1,000 sequences of multimodal data, encompassing high-resolution imagery from seven ring cameras, and two stereo cameras in addition to lidar point clouds, and 6-DOF map-aligned pose. Sequences contain 3D cuboid annotations for 26 object categories, all of which are sufficiently-sampled to support training and evaluation of 3D perception models. The Lidar Dataset contains 20,000 sequences of unlabeled lidar point clouds and map-aligned pose. This dataset is the largest ever collection of lidar sensor data and supports self-supervised learning and the emerging task of point cloud forecasting. Finally, the Motion Forecasting Dataset contains 250,000 scenarios mined for interesting and challenging interactions between the autonomous vehicle and other actors in each local scene. Models are tasked with the prediction of future motion for"scored actors"in each scenario and are provided with track histories that capture object location, heading, velocity, and category. In all three datasets, each scenario contains its own HD Map with 3D lane and crosswalk geometry - sourced from data captured in six distinct cities. We believe these datasets will support new and existing machine learning research problems in ways that existing datasets do not. All datasets are released under the CC BY-NC-SA 4.0 license.

One Million Scenes for Autonomous Driving: ONCE Dataset

Abstract: Current perception models in autonomous driving have become notorious for greatly relying on a mass of annotated data to cover unseen cases and address the long-tail problem. On the other hand, learning from unlabeled large-scale collected data and incrementally self-training powerful recognition models have received increasing attention and may become the solutions of next-generation industry-level powerful and robust perception models in autonomous driving. However, the research community generally suffered from data inadequacy of those essential real-world scene data, which hampers the future exploration of fully/semi/self-supervised methods for 3D perception. In this paper, we introduce the ONCE (One millioN sCenEs) dataset for 3D object detection in the autonomous driving scenario. The ONCE dataset consists of 1 million LiDAR scenes and 7 million corresponding camera images. The data is selected from 144 driving hours, which is 20x longer than the largest 3D autonomous driving dataset available (e.g. nuScenes and Waymo), and it is collected across a range of different areas, periods and weather conditions. To facilitate future research on exploiting unlabeled data for 3D detection, we additionally provide a benchmark in which we reproduce and evaluate a variety of self-supervised and semi-supervised methods on the ONCE dataset. We conduct extensive analyses on those methods and provide valuable observations on their performance related to the scale of used data. Data, code, and more information are available at https://once-for-auto-driving.github.io/index.html.

Detailed information of influential datasets for grid-centric perception. (*) The KITTI-based SemanticKITTI Dataset contains LiDAR scans with dense point-wise annotations. For additional dataset annotations, we indicate the number of bounding boxes. (**) Waymo Open Dataset provides dense labels for each LiDAR point with 23 classes and 3D bounding box labels with 4 classes. (-) indicates that no information is provided or the item does not exist in this dataset. SG: Singapore.

Grid-centric methods are mostly conducted on existing large-scale autonomous driving datasets with annotations of 3D object bounding boxes, LiDAR segmentation labels, annotations of 2D&3D lane and high definition maps. Most influential benchmarks for grid-centric perception include KITTI{{cite:4ae0beb}}, nuScenes{{cite:b68377b}}, Argoverse{{cite:6249ea5}}, Lyft L5{{cite:dc451bb}}, SemanticKITTI{{cite:bc6286b}}, KITTI-360{{cite:294e6c0}}, Waymo Open Dataset (WOD){{cite:2388825}} and Once dataset{{cite:9df79e6}}. Note that grid-centric perception is usually not a standard challenge for each dataset, therefore test sets are held out and most methods report their results on the validation set. Table. {{table:0cc04b79-18fb-45e9-8b2d-8624420b43cf}}  provides a summary of these benchmarks' information.


FinBERT: A Pretrained Language Model for Financial Communications

Abstract: Contextual pretrained language models, such as BERT (Devlin et al., 2019), have made significant breakthrough in various NLP tasks by training on large scale of unlabeled text re-sources.Financial sector also accumulates large amount of financial communication text.However, there is no pretrained finance specific language models available. In this work,we address the need by pretraining a financial domain specific BERT models, FinBERT, using a large scale of financial communication corpora. Experiments on three financial sentiment classification tasks confirm the advantage of FinBERT over generic domain BERT model. The code and pretrained models are available at this https URL. We hope this will be useful for practitioners and researchers working on financial NLP tasks.

A Benchmark for Lease Contract Review

Abstract: Extracting entities and other useful information from legal contracts is an important task whose automation can help legal professionals perform contract reviews more efficiently and reduce relevant risks. In this paper, we tackle the problem of detecting two different types of elements that play an important role in a contract review, namely entities and red flags. The latter are terms or sentences that indicate that there is some danger or other potentially problematic situation for one or more of the signing parties. We focus on supporting the review of lease agreements, a contract type that has received little attention in the legal information extraction literature, and we define the types of entities and red flags needed for that task. We release a new benchmark dataset of 179 lease agreement documents that we have manually annotated with the entities and red flags they contain, and which can be used to train and test relevant extraction algorithms. Finally, we release a new language model, called ALeaseBERT, pre-trained on this dataset and fine-tuned for the detection of the aforementioned elements, providing a baseline for further research

CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Abstract: We present CodeBERT, a bimodal pre-trained model for programming language (PL) and nat-ural language (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural language codesearch, code documentation generation, etc. We develop CodeBERT with Transformer-based neural architecture, and train it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us to utilize both bimodal data of NL-PL pairs and unimodal data, where the former provides input tokens for model training while the latter helps to learn better generators. We evaluate CodeBERT on two NL-PL applications by fine-tuning model parameters. Results show that CodeBERT achieves state-of-the-art performance on both natural language code search and code documentation generation tasks. Furthermore, to investigate what type of knowledge is learned in CodeBERT, we construct a dataset for NL-PL probing, and evaluate in a zero-shot setting where parameters of pre-trained models are fixed. Results show that CodeBERT performs better than previous pre-trained models on NL-PL probing.

Unified Pre-training for Program Understanding and Generation

Abstract: Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART’s effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., “if“ block inside an “else“ block is equivalent to “else if“ block) that are crucial to program semantics and thus excels even with limited annotations.

GraphCodeBERT: Pre-training Code Representations with Data Flow

Abstract: Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc. However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process. We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables. Such a semantic-level structure is neat and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efficient. We develop GraphCodeBERT based on Transformer. In addition to using the task of masked language modeling, we introduce two structure-aware pre-training tasks. One is to predict code structure edges, and the other is to align representations between source code and code structure. We implement the model in an efficient way with a graph-guided masked attention function to incorporate the code structure. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refinement. Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks. We further show that the model prefers structure-level attentions over token-level attentions in the task of code search.

CoTexT: Multi-task Learning with Code-Text Transformer

Abstract: We present CoTexT, a pre-trained, transformer-based encoder-decoder model that learns the representative context between natural language (NL) and programming language (PL). Using self-supervision, CoTexT is pre-trained on large programming language corpora to learn a general understanding of language and code. CoTexT supports downstream NL-PL tasks such as code summarizing/documentation, code generation, defect detection, and code debugging. We train CoTexT on different combinations of available PL corpus including both "bimodal" and "unimodal" data. Here, bimodal data is the combination of text and corresponding code snippets, whereas unimodal data is merely code snippets. We first evaluate CoTexT with multi-task learning: we perform Code Summarization on 6 different programming languages and Code Refinement on both small and medium size featured in the CodeXGLUE dataset. We further conduct extensive experiments to investigate CoTexT on other tasks within the CodeXGlue dataset, including Code Generation and Defect Detection. We consistently achieve SOTA results in these tasks, demonstrating the versatility of our models.

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

Abstract: Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems.

MathBERT: A Pre-Trained Model for Mathematical Formula Understanding

Abstract: Large-scale pre-trained models like BERT, have obtained a great success in various Natural Language Processing (NLP) tasks, while it is still a challenge to adapt them to the math-related tasks. Current pre-trained models neglect the structural features and the semantic correspondence between formula and its context. To address these issues, we propose a novel pre-trained model, namely \textbf{MathBERT}, which is jointly trained with mathematical formulas and their corresponding contexts. In addition, in order to further capture the semantic-level structural features of formulas, a new pre-training task is designed to predict the masked formula substructures extracted from the Operator Tree (OPT), which is the semantic structural representation of formulas. We conduct various experiments on three downstream tasks to evaluate the performance of MathBERT, including mathematical information retrieval, formula topic classification and formula headline generation. Experimental results demonstrate that MathBERT significantly outperforms existing methods on all those three tasks. Moreover, we qualitatively show that this pre-trained model effectively captures the semantic-level structural information of formulas. To the best of our knowledge, MathBERT is the first pre-trained model for mathematical formula understanding.

TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue

Abstract: The use of pre-trained language models has emerged as a promising direction for improving dialogue systems. However, the underlying difference of linguistic patterns between conversational data and general text makes the existing pre-trained language models not as effective as they have been shown to be. Recently, there are some pre-training approaches based on open-domain dialogues, leveraging large-scale social media data such as Twitter or Reddit. Pre-training for task-oriented dialogues, on the other hand, is rarely discussed because of the long-standing and crucial data scarcity problem. In this work, we combine nine English-based, human-human, multi-turn and publicly available task-oriented dialogue datasets to conduct language model pre-training. The experimental results show that our pre-trained task-oriented dialogue BERT (ToD-BERT) surpasses BERT and other strong baselines in four downstream task-oriented dialogue applications, including intention detection, dialogue state tracking, dialogue act prediction, and response selection. Moreover, in the simulated limited data experiments, we show that ToD-BERT has stronger few-shot capacity that can mitigate the data scarcity problem in task-oriented dialogues.

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Abstract: Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.

Publicly Available Clinical BERT Embeddings

Abstract: Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for generic clinical text and another for discharge summaries specifically. We demonstrate that using a domain-specific model yields performance improvements on three common clinical NLP tasks as compared to nonspecific embeddings. These domain-specific models are not as performant on two clinical de-identification tasks, and argue that this is a natural consequence of the differences between de-identified source text and synthetically non de-identified task text.

Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

Abstract: Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We make the datasets, pre-trained models, and codes publicly available at https://github.com/ncbi-nlp/BLUE_Benchmark.

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Abstract: Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly-available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition (NER). To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB.

Summary of domain-specific T-PTLMs.

Following the success of T-PTLMs in general domain, T-PTLMs in specific domains like Finance {{cite:5a0d2ac}}, Legal {{cite:eba8596}}, {{cite:39e3f3f}}, News {{cite:e9d21ed}}, Programming {{cite:30830e0}}, {{cite:9eeb10f}}, {{cite:aa8207e}}, {{cite:f77ed90}}, {{cite:35f3d69}}, Dialogue {{cite:217f882}}, Networking {{cite:1b7e6c8}}, Academic {{cite:7630bb8}}, {{cite:cf23801}}, {{cite:34a6fe6}} and Biomedical {{cite:b494b3a}}, {{cite:82bb449}}, {{cite:29bf417}}, {{cite:915bcea}} have been developed (refer Table {{table:0f0297ec-47d1-4b94-b63c-00cd414321c0}}  for a brief summary). Models like BERT, RoBERTa, BART, and T5 are pretrained on general domain text. For a model to perform well on domain-specific datasets the model should have enough domain knowledge {{cite:b494b3a}}, {{cite:5a0d2ac}}. These general domain models can not acquire enough domain knowledge just through fine-tuning. As a result, the performance of these models on domain-specific datasets is limited {{cite:b494b3a}}, {{cite:5a0d2ac}}. The initial trend to develop domain-specific models is using continual pretraining i.e., initialize the model with any of the existing general domain models and further pretrain on a domain-specific corpus. For example, BioBERT {{cite:b494b3a}} is the first domain-specific BERT model developed using continual pretraining. Following BioBERT in biomedical domain, models like AleaseBERT {{cite:eba8596}}, RoBERTa-News {{cite:e9d21ed}}, GraphCodeBERT {{cite:f77ed90}}, CoText {{cite:35f3d69}}, CodeGPT-adapted {{cite:30830e0}}, NetBERT {{cite:1b7e6c8}}, MathBERT {{cite:34a6fe6}}, TOD-BERT {{cite:217f882}}, ClinicalBERT {{cite:82bb449}} and BluBERT {{cite:915bcea}} have been developed using continual pretraining.


Mashable {{cite:9c8f093}}, Verge {{cite:2a1b886}}, Business Insider {{cite:55dd4cd}}

Boing Boing {{cite:00f9690}}, DigitalTrends {{cite:7f4ff5f}}

Quartz {{cite:9f088a6}}, Vice {{cite:1e421bd}}

Fooling Automated Surveillance Cameras: Adversarial Patches to Attack Person Detection

Abstract: Adversarial attacks on machine learning models have seen increasing interest in the past years. By making only subtle changes to the input of a convolutional neural network, the output of the network can be swayed to output a completely different result. The first attacks did this by changing pixel values of an input image slightly to fool a classifier to output the wrong class. Other approaches have tried to learn "patches" that can be applied to an object to fool detectors and classifiers. Some of these approaches have also shown that these attacks are feasible in the real-world, i.e. by modifying an object and filming it with a video camera. However, all of these approaches target classes that contain almost no intra-class variety (e.g. stop signs). The known structure of the object is then used to generate an adversarial patch on top of it. In this paper, we present an approach to generate adversarial patches to targets with lots of intra-class variety, namely persons. The goal is to generate a patch that is able successfully hide a person from a person detector. An attack that could for instance be used maliciously to circumvent surveillance systems, intruders can sneak around undetected by holding a small cardboard plate in front of their body aimed towards the surveilance camera. From our results we can see that our system is able significantly lower the accuracy of a person detector. Our approach also functions well in real-life scenarios where the patch is filmed by a camera. To the best of our knowledge we are the first to attempt this kind of attack on targets with a high level of intra-class variety like persons.

Invisible Mask: Practical Attacks on Face Recognition with Infrared

Abstract: Accurate face recognition techniques make a series of critical applications possible: policemen could employ it to retrieve criminals' faces from surveillance video streams; cross boarder travelers could pass a face authentication inspection line without the involvement of officers. Nonetheless, when public security heavily relies on such intelligent systems, the designers should deliberately consider the emerging attacks aiming at misleading those systems employing face recognition. We propose a kind of brand new attack against face recognition systems, which is realized by illuminating the subject using infrared according to the adversarial examples worked out by our algorithm, thus face recognition systems can be bypassed or misled while simultaneously the infrared perturbations cannot be observed by raw eyes. Through launching this kind of attack, an attacker not only can dodge surveillance cameras. More importantly, he can impersonate his target victim and pass the face authentication system, if only the victim's photo is acquired by the attacker. Again, the attack is totally unobservable by nearby people, because not only the light is invisible, but also the device we made to launch the attack is small enough. According to our study on a large dataset, attackers have a very high success rate with a over 70\% success rate for finding such an adversarial example that can be implemented by infrared. To the best of our knowledge, our work is the first one to shed light on the severity of threat resulted from infrared adversarial examples against face recognition.

AdvHat: Real-World Adversarial Attack on ArcFace Face ID System

Abstract: In this paper we propose a novel easily reproducible technique to attack the best public Face ID system ArcFace in different shooting conditions. To create an attack, we print the rectangular paper sticker on a common color printer and put it on the hat. The adversarial sticker is prepared with a novel algorithm for off-plane transformations of the image which imitates sticker location on the hat. Such an approach confuses the state-of-the-art public Face ID model LResNet100E-IR, ArcFace@ms1m-refine-v2 and is transferable to other Face ID models.

A General Framework for Adversarial Examples with Objectives

Abstract: Images perturbed subtly to be misclassified by neural networks, called adversarial examples, have emerged as a technically deep challenge and an important concern for several application domains. Most research on adversarial examples takes as its only constraint that the perturbed images are similar to the originals. However, real-world application of these ideas often requires the examples to satisfy additional objectives, which are typically enforced through custom modifications of the perturbation process. In this article, we propose adversarial generative nets (AGNs), a general methodology to train a generator neural network to emit adversarial examples satisfying desired objectives. We demonstrate the ability of AGNs to accommodate a wide range of objectives, including imprecise ones difficult to model, in two application domains. In particular, we demonstrate physical adversarial examples—eyeglass frames designed to fool face recognition—with better robustness, inconspicuousness, and scalability than previous approaches, as well as a new attack to fool a handwritten-digit classifier.

Media Coverage

Advances in machine learning (ML) techniques have enabled computer vision systems such as facial and person recognition to erode expectations of privacy and anonymity online and in person. These systems are already being employed to track dissidents and identify protesters {{cite:c638ad4}}, {{cite:0bfdcc0}}. Adversarial ML attacks are one way for individuals to combat surveillance. Unfortunately, the limited adversarial ML attacks that exist are not yet ready for the “real world” {{cite:c803af4}}. In this paper, we turn a critical lens on the adequacy and representativeness of physical domain testing for various adversarial ML attacks against computer vision systems involving human subjects. Many papers that use physical adversarial attacks characterize themselves as “real world”or “practical” {{cite:e03a784}}, {{cite:1e18e12}}, {{cite:487704f}} and many receive glowing write ups in popular news outlets (see Table {{table:0dbe8e1c-9754-43ed-ac3e-1a0ae44b720a}} ). In practice, however, we found that physical or real-world testing was minimal, papers provided few details about testing subjects, and testing was often conducted as an afterthought or as a demonstration. The hype around adversarial ML attacks have tangible implications: multiple authors of this paper have been approached by advocates who wish to use adversarial machine learning techniques in high risk, real world contexts.


GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

Abstract: Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique to trade off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples. Samples from a 3.5 billion parameter text-conditional diffusion model using classifier-free guidance are favored by human evaluators to those from DALL-E, even when the latter uses expensive CLIP reranking. Additionally, we find that our models can be fine-tuned to perform image inpainting, enabling powerful text-driven image editing. We train a smaller model on a filtered dataset and release the code and weights at https://github.com/openai/glide-text2im.

High-Resolution Image Synthesis with Latent Diffusion Models

Abstract: By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve new state of the art scores for image inpainting and class-conditional image synthesis and highly competitive performance on various tasks, including unconditional image generation, text-to-image synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs.

Scalable Diffusion Models with Transformers

Abstract: We explore a new class of diffusion models based on the transformer architecture. We train latent diffusion models of images, replacing the commonly-used U-Net backbone with a transformer that operates on latent patches. We analyze the scalability of our Diffusion Transformers (DiTs) through the lens of forward pass complexity as measured by Gflops. We find that DiTs with higher Gflops—through increased transformer depth/width or increased number of input tokens—consistently have lower FID. In addition to possessing good scalability properties, our largest DiT-XL/2 models outperform all prior diffusion models on the class-conditional ImageNet 512×512 and 256×256 benchmarks, achieving a state-of-the-art FID of 2.27 on the latter.

Comparison of three state-of-the-art diffusion models on ImageNet-1K using their default hyper-parameters to generate synthetic images. "#Syn Images" represents the total number of synthetic images. We use the pre-trained ResNet18 as the teacher to train the vanilla ResNet18 student model.

where {{formula:7aa1ae25-ad5d-4fd7-81c6-c9595f79e800}}  is the sampled noise predicted at timestep {{formula:27987078-1106-48e7-9ad8-1da69e8b8770}}  with condition {{formula:dacbdabd-d891-44b0-a072-3977b335536d}} , and {{formula:fdcc8c0c-88c2-4aef-8f8f-7c1d05911156}}  is the unconditional predicted noise.
Here, the hyperparameter {{formula:435c96db-173d-45d9-a12f-991dd5846103}}  is used to adjust the scale of the guidance, with {{formula:7a57af60-6153-4924-a012-68ac9c57663a}} =1 indicating that no classifier-free guidance is employed. Additionally, {{formula:788e808f-33ec-44b8-bfe6-04b58484354f}}  represents a trainable "null" condition.
Based on the performance comparison between DiT, Glide, and SD in Table {{table:cf20d53b-f3c6-4b34-af3f-45fc96766ea9}} , we default to use DiT as our synthetic data generator for knowledge distillation in this study. Data generation and student training details are presented in the supplementary material.


PieAPP: Perceptual Image-Error Assessment Through Pairwise Preference

Abstract: The ability to estimate the perceptual error between images is an important problem in computer vision with many applications. Although it has been studied extensively, however, no method currently exists that can robustly predict visual differences like humans. Some previous approaches used hand-coded models, but they fail to model the complexity of the human visual system. Others used machine learning to train models on human-labeled datasets, but creating large, high-quality datasets is difficult because people are unable to assign consistent error labels to distorted images. In this paper, we present a new learning-based method that is the first to predict perceptual image error like human observers. Since it is much easier for people to compare two given images and identify the one more similar to a reference than to assign quality scores to each, we propose a new, large-scale dataset labeled with the probability that humans will prefer one image over another. We then train a deep-learning model using a novel, pairwise-learning framework to predict the preference of one distorted image over the other. Our key observation is that our trained network can then be used separately with only one distorted image and a reference to predict its perceptual error, without ever being trained on explicit human perceptual-error labels. The perceptual error estimated by our new metric, PieAPP, is well-correlated with human opinion. Furthermore, it significantly outperforms existing algorithms, beating the state-of-the-art by almost 3Ã— on our test set in terms of binary error rate, while also generalizing to new kinds of distortions, unlike previous learning-based methods.

DeepFL-IQA: Weak Supervision for Deep IQA Feature Learning

Abstract: Multi-level deep-features have been driving state-of-the-art methods for aesthetics and image quality assessment (IQA). However, most IQA benchmarks are comprised of artificially distorted images, for which features derived from ImageNet under-perform. We propose a new IQA dataset and a weakly supervised feature learning approach to train features more suitable for IQA of artificially distorted images. The dataset, KADIS-700k, is far more extensive than similar works, consisting of 140,000 pristine images, 25 distortions types, totaling 700k distorted versions. Our weakly supervised feature learning is designed as a multi-task learning type training, using eleven existing full-reference IQA metrics as proxies for differential mean opinion scores. We also introduce a benchmark database, KADID-10k, of artificially degraded images, each subjectively annotated by 30 crowd workers. We make use of our derived image feature vectors for (no-reference) image quality assessment by training and testing a shallow regression network on this database and five other benchmark IQA databases. Our method, termed DeepFL-IQA, performs better than other feature-based no-reference IQA methods and also better than all tested full-reference IQA methods on KADID-10k. For the other five benchmark IQA databases, DeepFL-IQA matches the performance of the best existing end-to-end deep learning-based methods on average.

Comparison of the available IQA datasets.

We assess the performance of our method using the common major IQA datasets: LIVE {{cite:6d6cdde}}, CSIQ {{cite:b0b6ea1}}, TID2013 {{cite:9a65c3b}}, and KADID-10k {{cite:0a785b2}}. PieAPP {{cite:af9a1aa}} and KADIS-700k {{cite:5aa8c47}} datasets are used for pre-training our model. Table {{table:dac56e11-9a61-4b89-87df-5a402dedbfd2}}  compares the listed datasets in more detail. Note that we omit the BAPPS dataset {{cite:2038360}} as it targets perceptual patch similarity and JND as opposed to image quality.