Open Graph Benchmark: Datasets for Machine Learning on Graphs

Abstract: We present the Open Graph Benchmark (OGB), a diverse set of challenging and realistic benchmark datasets to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale (up to 100+ million nodes and 1+ billion edges), encompass multiple important graph ML tasks, and cover a diverse range of domains, ranging from social and information networks to biological networks, molecular graphs, source code ASTs, and knowledge graphs. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics. In addition to building the datasets, we also perform extensive benchmark experiments for each dataset. Our experiments suggest that OGB datasets present significant challenges of scalability to large-scale graphs and out-of-distribution generalization under realistic data splits, indicating fruitful opportunities for future research. Finally, OGB provides an automated end-to-end graph ML pipeline that simplifies and standardizes the process of graph data loading, experimental setup, and model evaluation. OGB will be regularly updated and welcomes inputs from the community. OGB datasets as well as data loaders, evaluation scripts, baseline code, and leaderboards are publicly available at this https URL .

Heterogeneous Graph Representation Learning With Relation Awareness

Abstract: Representation learning on heterogeneous graphs aims to obtain meaningful node representations to facilitate various downstream tasks, such as node classification and link prediction. Existing heterogeneous graph learning methods are primarily developed by following the propagation mechanism of node representations. There are few efforts on studying the role of relations for improving the learning of more fine-grained node representations. Indeed, it is important to collaboratively learn the semantic representations of relations and discern node representations with respect to different relation types. To this end, in this paper, we propose a Relation-aware Heterogeneous Graph Neural Network, namely R-HGNN, to learn node representations on heterogeneous graphs at a fine-grained level by considering relation-aware characteristics. Specifically, a dedicated graph convolution component is first designed to learn unique node representations from each relation-specific graph separately. Then, a cross-relation message passing module is developed to improve the interactions of node representations across different relations. Also, the relation representations are learned in a layer-wise manner to capture relation semantics, which are used to guide the node representation learning process. Moreover, a semantic fusing module is presented to aggregate relation-aware node representations into a compact representation with the learned relation representations. Finally, we conduct extensive experiments on a variety of graph learning tasks, and experimental results demonstrate that our approach consistently outperforms existing methods among all the tasks.

Statistics of the datasets.

The task type on all the datasets is multi-class classification as each node belongs to exactly one label. All the datasets are split by the publication dates of papers and we follow the same data splits in {{cite:277f57b}} and {{cite:dd9cd68}}. We define the homophily as the fraction of edges in a graph whose endpoints have the same label, that is, {{formula:bcdcc3ba-202e-4e8c-9508-ba9cd8439941}} . For ogbn-mag and oag-venue, we calculate the graph homophily on the subgraph that only contains papers.
Statistics of the datasets are shown in Table {{table:8dd86afc-6d22-4816-8612-932ac07fb61e}} .


A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Abstract: This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. recognizing textual entailment), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by offering data from ten distinct genres of written and spoken English, making it possible to evaluate systems on nearly the full complexity of the language, while supplying an explicit setting for evaluating cross-genre domain adaptation. In addition, an evaluation using existing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement.

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Abstract:
We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research.
The dataset is freely available at this https URL

Performance after different training stages in MoMo. Stages 2 and 3 bring in considerable performance gains for vision and multimodal tasks respectively.

Multi-Stage Training. Earlier work {{cite:5c2231c}} {{cite:c317669}} reveal that initializing multimodal model with vision weights is more effective than initializing it with pre-trained text weights. On similar lines, we first train MoMo with unimodal image data before learning unimodal text and multimodal image-text representations. The performance of MoMo after each stage on various tasks is reported in Table {{table:b8446fb8-f1f9-439a-9556-ba05673ae6d8}} . After stage 1, the model achieves feasible image classification performance, while results on other tasks remain suboptimal. Stage 2 helps the model to learn language-related information and thus improves the performance on language tasks. Stage 3 further enhances the model capacity on multimodal tasks. We note that the performance on image classification degrades after Stage 2. We attribute this drop to the shared encoder design on learning different modalities. However, we observe that this drop is recovered and further improved during Stage 3, while language performance is retained. During stage 3, we also evaluated a variant that includes image-only training (along with text-only and multimodal). However, this didn't show significant differences on downstream tasks and hence we decided to save compute and skip this additional objective.


HateBERT: Retraining BERT for Abusive Language Detection in English

Abstract: We introduce HateBERT, a re-trained BERT model for abusive language detection in English. The model was trained on RAL-E, a large-scale dataset of Reddit comments in English from communities banned for being offensive, abusive, or hateful that we have curated and made available to the public. We present the results of a detailed comparison between a general pre-trained language model and the retrained version on three English datasets for offensive, abusive language and hate speech detection tasks. In all datasets, HateBERT outperforms the corresponding general BERT model. We also discuss a battery of experiments comparing the portability of the fine-tuned models across the datasets, suggesting that portability is affected by compatibility of the annotated phenomena.

TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification

Abstract: The experimental landscape in natural language processing for social media is too fragmented. Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction. Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domain-specific data. In this paper, we propose a new evaluation framework (TweetEval) consisting of seven heterogeneous Twitter-specific classification tasks. We also provide a strong set of baselines as starting point, and compare different language modeling pre-training strategies. Our initial experiments show the effectiveness of starting off with existing pre-trained generic language models, and continue training them on Twitter corpora.

BERTweet: A pre-trained language model for English Tweets

Abstract: We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having the same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al., 2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification. We release BERTweet under the MIT License to facilitate future research and applications on Tweet data. Our BERTweet is available at https://github.com/VinAIResearch/BERTweet

COVID-Twitter-BERT: A natural language processing model to analyse COVID-19 content on Twitter

Abstract: Introduction This study presents COVID-Twitter-BERT (CT-BERT), a transformer-based model that is pre-trained on a large corpus of COVID-19 related Twitter messages. CT-BERT is specifically designed to be used on COVID-19 content, particularly from social media, and can be utilized for various natural language processing tasks such as classification, question-answering, and chatbots. This paper aims to evaluate the performance of CT-BERT on different classification datasets and compare it with BERT-LARGE, its base model. Methods The study utilizes CT-BERT, which is pre-trained on a large corpus of COVID-19 related Twitter messages. The authors evaluated the performance of CT-BERT on five different classification datasets, including one in the target domain. The model's performance is compared to its base model, BERT-LARGE, to measure the marginal improvement. The authors also provide detailed information on the training process and the technical specifications of the model. Results The results indicate that CT-BERT outperforms BERT-LARGE with a marginal improvement of 10-30% on all five classification datasets. The largest improvements are observed in the target domain. The authors provide detailed performance metrics and discuss the significance of these results. Discussion The study demonstrates the potential of pre-trained transformer models, such as CT-BERT, for COVID-19 related natural language processing tasks. The results indicate that CT-BERT can improve the classification performance on COVID-19 related content, especially on social media. These findings have important implications for various applications, such as monitoring public sentiment and developing chatbots to provide COVID-19 related information. The study also highlights the importance of using domain-specific pre-trained models for specific natural language processing tasks. Overall, this work provides a valuable contribution to the development of COVID-19 related NLP models.

Summary of social-media based T-PTLMs.

T-PTLMs like BERT and RoBERTa are pretrained on formal text. As social media text is highly informal in nature with a lot of noise in the form of irregular grammar, slang words, and non-standard abbreviations, these models have limited performance on social media datasets {{cite:a9a10e5}}, {{cite:8d6ee38}}, {{cite:c1a208f}}, {{cite:cd64436}} . Researchers working at the intersection of social media and NLP have developed social media-specific T-PTLMs either by training from scratch {{cite:8d6ee38}} or continual pretraining {{cite:a9a10e5}}, {{cite:c1a208f}}, {{cite:cd64436}}, {{cite:0f059d1}}, {{cite:28212cb}}, {{cite:62d1f64}} and a summary of these models is presented in Table {{table:5727c16c-c133-4ee5-aca6-168dfc458fd3}} . Except for Bertweet {{cite:8d6ee38}}, all other social media-based T-PTLMs are developed by continual pretraining. Training from scratch is effective only when the pretraining corpus consists of a large number of tweets. Otherwise, continual pretraining is recommended. For example, BERTweet {{cite:8d6ee38}} is pretrained from scratch using 850M tweets. Barbieri et al. {{cite:cd64436}} showed that RoBERTa model trained from scratch on tweets achieved less performance compared to RoBERTa model adapted to social media by continual pretraining. This is because of using just 60M tweets for pretraining. Different from other social media-based T-PTLMs which are developed using commonly used pretraining tasks like MLM and NSP, BERT-SentiX {{cite:62d1f64}} is obtained by continual pretraining on user reviews using four novel sentiment aware pretraining tasks.


Robust 3D Hand Pose Estimation in Single Depth Images: From Single-View CNN to Multi-View CNNs

Abstract: Articulated hand pose estimation plays an important role in human-computer interaction. Despite the recent progress, the accuracy of existing methods is still not satisfactory, partially due to the difficulty of embedded high-dimensional and non-linear regression problem. Different from the existing discriminative methods that regress for the hand pose with a single depth image, we propose to first project the query depth image onto three orthogonal planes and utilize these multi-view projections to regress for 2D heat-maps which estimate the joint positions on each plane. These multi-view heat-maps are then fused to produce final 3D hand pose estimation with learned pose priors. Experiments show that the proposed method largely outperforms state-of-the-art on a challenging dataset. Moreover, a cross-dataset experiment also demonstrates the good generalization ability of the proposed method.

Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose Estimation

Abstract: State-of-the-art methods for 3D hand pose estimation from depth images require large amounts of annotated training data. We propose modelling the statistical relationship of 3D hand poses and corresponding depth images using two deep generative models with a shared latent space. By design, our architecture allows for learning from unlabeled image data in a semi-supervised manner. Assuming a one-to-one mapping between a pose and a depth map, any given point in the shared latent space can be projected into both a hand pose or into a corresponding depth map. Regressing the hand pose can then be done by learning a discriminator to estimate the posterior of the latent pose given some depth map. To prevent over-fitting and to better exploit unlabeled depth maps, the generator and discriminator are trained jointly. At each iteration, the generator is updated with the back-propagated gradient from the discriminator to synthesize realistic depth maps of the articulated hand, while the discriminator benefits from an augmented training set of synthesized samples and unlabeled depth maps. The proposed discriminator network architecture is highly efficient and runs at 90fps on the CPU with accuracies comparable or better than state-of-art on 3 publicly available benchmarks.

Comparison to previous work on MSRA15. Ours yields the lowest error and improves {{cite:aedfab0}} by 1.27mm (relative {{formula:5f8df112-dffb-445f-900d-77860551e884}} ).

Comparisons to the state of the art We compare our method to previous works in Table {{table:fcfdd3d4-fa17-4eea-9c28-eed7d62a7198}}  with the mean 3D joint error metric. Our method achieves a 5.96mm mean 3D joint error and outperforms the previous state-of-the-art result of {{formula:3bf8d2dc-2d35-4198-9fc2-003b966d1672}}  {{cite:aedfab0}} by a large margin (17.6% relative improvement). Please see the supplement for more comparisons using the PCF metric.


Train the generator to produce samples that are as close as possible to the real data distribution

Predict the relative positions of random patch pairs

Predict the rotation angle of the random rotated image

Store negative samples in a queue and perform momentum updates to the key code.

Contrastive learning for online clustering

Average a teacher network with a predictor on top of a teacher encoder

Explore the simplest contrasting SSL designs

Masked Autoencoders Are Scalable Vision Learners

Abstract: This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior.

Wasserstein GAN

Abstract: We introduce a new algorithm named WGAN, an alternative to traditional GAN training. In this new model, we show that we can improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches. Furthermore, we show that the corresponding optimization problem is sound, and provide extensive theoretical work highlighting the deep connections to other distances between distributions.

Unsupervised Visual Representation Learning by Context Prediction

Abstract: This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the position of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recognize objects and their parts. We demonstrate that the feature representation learned using this within-image context indeed captures visual similarity across images. For example, this representation allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset. Furthermore, we show that the learned ConvNet can be used in the R-CNN framework [19] and provides a significant boost over a randomly-initialized ConvNet, resulting in state-of-the-art performance among algorithms which use only Pascal-provided training set annotations.

Unsupervised Representation Learning by Predicting Image Rotations

Abstract: Over the last years, deep convolutional neural networks (ConvNets) have transformed the field of computer vision thanks to their unparalleled capacity to learn high level semantic image features. However, in order to successfully learn those features, they usually require massive amounts of manually labeled data, which is both expensive and impractical to scale. Therefore, unsupervised semantic feature learning, i.e., learning without requiring manual annotation effort, is of crucial importance in order to successfully harvest the vast amount of visual data that are available today. In our work we propose to learn image features by training ConvNets to recognize the 2d rotation that is applied to the image that it gets as input. We demonstrate both qualitatively and quantitatively that this apparently simple task actually provides a very powerful supervisory signal for semantic feature learning. We exhaustively evaluate our method in various unsupervised feature learning benchmarks and we exhibit in all of them state-of-the-art performance. Specifically, our results on those benchmarks demonstrate dramatic improvements w.r.t. prior state-of-the-art approaches in unsupervised representation learning and thus significantly close the gap with supervised feature learning. For instance, in PASCAL VOC 2007 detection task our unsupervised pre-trained AlexNet model achieves the state-of-the-art (among unsupervised methods) mAP of 54.4% that is only 2.4 points lower from the supervised case. We get similarly striking results when we transfer our unsupervised learned features on various other tasks, such as ImageNet classification, PASCAL classification, PASCAL segmentation, and CIFAR-10 classification. The code and models of our paper will be published on: this https URL .

Improved Baselines with Momentum Contrastive Learning

Abstract: Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing them in the MoCo framework. With simple modifications to MoCo---namely, using an MLP projection head and more data augmentation---we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public.

Emerging Properties in Self-Supervised Vision Transformers

Abstract: In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) [16] that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder [26], multi-crop training [9], and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Abstract: We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.

Exploring Simple Siamese Representation Learning

Abstract: Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code is made available.1

A representative collection of pretext tasks in the existing SSL methods.

Self-supervised learning on remote sensing (SSL)

There are two common strategies for pretext design to achieve different objectives: 1) Generative-based pretext task that reconstructs the input data (Such as discriminating images created from distortion {{cite:d14c284}}) {{formula:7afafdfa-b9a9-43c9-b9ef-63591b13abf7}} , or predicts a label {{formula:3bc465cd-8ca9-4d9a-a5ea-fc27510589e8}}  that is self-generated from context and data augmentation {{formula:99d35af6-d42e-4fd2-b1da-d4dd1fd4b6fa}} , and 2) Contrastive learning {{cite:d1a8c4e}} based pretext task that contrasts inputs {{formula:24ee1598-cb7e-47c7-8e2d-288c138a716d}}  and {{formula:b170c984-9b13-4335-8318-1972fae02c2f}}  that have similar meanings (for example, the encoded features of two different views of the same image should match {{cite:d1202f6}}, {{cite:02b64be}}) {{formula:76ff0c8e-c5f9-454b-ad5f-4dff53ff70e5}} . Table. {{table:2d45974d-171a-4301-b3e6-3134ef3dbbbc}}  summarises the representative approaches for different types of pretext tasks.


Reducing the operational time using energy management system

Peak average ratio, energy consumed, generated and load pricing

Accuracy of load, and price of electricity

Game Theoretical Energy Management with Storage Capacity Optimization and Photo-Voltaic Cell Generated Power Forecasting in Micro Grid

Abstract: In order to ensure optimal and secure functionality of Micro Grid (MG), energy management system plays vital role in managing multiple electrical load and distributed energy technologies. With the evolution of Smart Grids (SG), energy generation system that includes renewable resources is introduced in MG. This work focuses on coordinated energy management of traditional and renewable resources. Users and MG with storage capacity is taken into account to perform energy management efficiently. First of all, two stage Stackelberg game is formulated. Every player in game theory tries to increase its payoff and also ensures user comfort and system reliability. In the next step, two forecasting techniques are proposed in order to forecast Photo Voltaic Cell (PVC) generation for announcing optimal prices. Furthermore, existence and uniqueness of Nash Equilibrium (NE) of energy management algorithm are also proved. In simulation, results clearly show that proposed game theoretic approach along with storage capacity optimization and forecasting techniques give benefit to both players, i.e., users and MG. The proposed technique Gray wolf optimized Auto Regressive Integrated Moving Average (GARIMA) gives 40% better result and Cuckoo Search Auto Regressive Integrated Moving Average (CARIMA) gives 30% better results as compared to existing techniques.

Short-Term Electric Load and Price Forecasting Using Enhanced Extreme Learning Machine Optimization in Smart Grids

Abstract: A Smart Grid (SG) is a modernized grid to provide efficient, reliable and economic energy to the consumers. Energy is the most important resource in the world. An efficient energy distribution is required as smart devices are increasing dramatically. The forecasting of electricity consumption is supposed to be a major constituent to enhance the performance of SG. Various learning algorithms have been proposed to solve the forecasting problem. The sole purpose of this work is to predict the price and load efficiently. The first technique is Enhanced Logistic Regression (ELR) and the second technique is Enhanced Recurrent Extreme Learning Machine (ERELM). ELR is an enhanced form of Logistic Regression (LR), whereas, ERELM optimizes weights and biases using a Grey Wolf Optimizer (GWO). Classification and Regression Tree (CART), Relief-F and Recursive Feature Elimination (RFE) are used for feature selection and extraction. On the basis of selected features, classification is performed using ELR. Cross validation is done for ERELM using Monte Carlo and K-Fold methods. The simulations are performed on two different datasets. The first dataset, i.e., UMass Electric Dataset is multi-variate while the second dataset, i.e., UCI Dataset is uni-variate. The first proposed model performed better with UMass Electric Dataset than UCI Dataset and the accuracy of second model is better with UCI than UMass. The prediction accuracy is analyzed on the basis of four different performance metrics: Mean Absolute Percentage Error (MAPE), Mean Absolute Error (MAE), Mean Square Error (MSE) and Root Mean Square Error (RMSE). The proposed techniques are then compared with four benchmark schemes. The comparison is done to verify the adaptivity of the proposed techniques. The simulation results show that the proposed techniques outperformed benchmark schemes. The proposed techniques efficiently increased the prediction accuracy of load and price. However, the computational time is increased in both scenarios. ELR achieved almost 5% better results than Convolutional Neural Network (CNN) and almost 3% than LR. While, ERELM achieved almost 6% better results than ELM and almost 5% than RELM. However, the computational time is almost 20% increased with ELR and 50% with ERELM. Scalability is also addressed for the proposed techniques using half-yearly and yearly datasets. Simulation results show that ELR gives 5% better results while, ERELM gives 6% better results when used for yearly dataset.

Optimization using GWO in Smart Grids

RQ3: What are the objective functions/ parameters used for obtaining an optimal energy management/cost-efficient system in Smart Grids?

Tables {{table:75a0b27e-7767-4867-adbc-714b7c73368d}} , {{table:a0ded1cf-f5a5-4f8d-bc0d-cc68a6f6bfd4}} , and {{table:0326d74b-216e-4d84-a918-12b9c9a4bfe2}}  present the objective functions and parameters used for obtaining cost efficient optimal energy management using GA, GWO, and PSO, respectively. Moreover, Table {{table:0326d74b-216e-4d84-a918-12b9c9a4bfe2}}  elaborates the articles that uses PSO to reduce the cost of energy by using energy management techniques in SGs. The collected research articles conclude that energy cost can be minimized by scheduling/managing the loads and optimizing the power losses, energy costs, and utilizing extra energy storage (if used). However, RMSE and peak average ratio are the least used parameters for optimizing the costs with energy management. We illustrate in Figure {{figure:4675532f-5276-4772-baba-1f3c2af20d47}}  the research databases (and the number of articles per each database) for the included articles in addressing RQ3. Figure {{figure:45d4f2ec-e990-4745-aa6b-ebe51ec79f4f}}  shows details of parameters used by the articles included in this survey. The figure concludes that cost functions are most widely used to optimize the functions. “Others” parameter in Figure {{figure:45d4f2ec-e990-4745-aa6b-ebe51ec79f4f}}  include time stamps, time functions, current, accuracy of load, droop parameters, emissions, transformer and capacitor parameters, latency, etc. As illustrated in Figure {{figure:45d4f2ec-e990-4745-aa6b-ebe51ec79f4f}} , the least used parameters to optimize the cost and energy management issues in SGs are Gain and Throughput.


Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST)

Spectro-Photometer for the History of the Universe, Epoch of Reionization, and Ices Explorer (SPHEREx)

LSST: from Science Drivers to Reference Design and Anticipated Data Products

Abstract: (Abridged) We describe here the most ambitious survey currently planned in the optical, the Large Synoptic Survey Telescope (LSST). A vast array of science will be enabled by a single wide-deep-fast sky survey, and LSST will have unique survey capability in the faint time domain. The LSST design is driven by four main science themes: probing dark energy and dark matter, taking an inventory of the Solar System, exploring the transient optical sky, and mapping the Milky Way. LSST will be a wide-field ground-based system sited at Cerro Pach\'{o}n in northern Chile. The telescope will have an 8.4 m (6.5 m effective) primary mirror, a 9.6 deg$^2$ field of view, and a 3.2 Gigapixel camera. The standard observing sequence will consist of pairs of 15-second exposures in a given field, with two such visits in each pointing in a given night. With these repeats, the LSST system is capable of imaging about 10,000 square degrees of sky in a single filter in three nights. The typical 5$\sigma$ point-source depth in a single visit in $r$ will be $\sim 24.5$ (AB). The project is in the construction phase and will begin regular survey operations by 2022. The survey area will be contained within 30,000 deg$^2$ with $\delta<+34.5^\circ$, and will be imaged multiple times in six bands, $ugrizy$, covering the wavelength range 320--1050 nm. About 90\% of the observing time will be devoted to a deep-wide-fast survey mode which will uniformly observe a 18,000 deg$^2$ region about 800 times (summed over all six bands) during the anticipated 10 years of operations, and yield a coadded map to $r\sim27.5$. The remaining 10\% of the observing time will be allocated to projects such as a Very Deep and Fast time domain survey. The goal is to make LSST data products, including a relational database of about 32 trillion observations of 40 billion objects, available to the public and scientists around the world.

The Dark Energy Spectroscopic Instrument (DESI)

Abstract: We present the status of the Dark Energy Spectroscopic Instrument (DESI) and its plans and opportunities for the coming decade. DESI construction and its initial five years of operations are an approved experiment of the US Department of Energy and is summarized here as context for the Astro2020 panel. Beyond 2025, DESI will require new funding to continue operations. We expect that DESI will remain one of the world's best facilities for wide-field spectroscopy throughout the decade. More about the DESI instrument and survey can be found at https://www.desi.lbl.gov.

Cosmology with the SPHEREX All-Sky Spectral Survey

Abstract: SPHEREx (Spectro-Photometer for the History of the Universe, Epoch of Reionization, and Ices Explorer) ( http://spherex.caltech.edu ) is a proposed all-sky spectroscopic survey satellite designed to address all three science goals in NASA's Astrophysics Division: probe the origin and destiny of our Universe; explore whether planets around other stars could harbor life; and explore the origin and evolution of galaxies. SPHEREx will scan a series of Linear Variable Filters systematically across the entire sky. The SPHEREx data set will contain R=40 spectra fir 0.75$<\lambda<$4.1$\mu$m and R=150 spectra for 4.1$<\lambda<$4.8$\mu$m for every 6.2 arc second pixel over the entire-sky. In this paper, we detail the extra-galactic and cosmological studies SPHEREx will enable and present detailed systematic effect evaluations. We also outline the Ice and Galaxy Evolution Investigations.

Packed Ultra-wideband Mapping Array (PUMA): A Radio Telescope for Cosmology and Transients

Abstract: PUMA is a proposal for an ultra-wideband, low-resolution and transit interferometric radio telescope operating at $200-1100\,\mathrm{MHz}$. Its design is driven by six science goals which span three science themes: the physics of dark energy (measuring the expansion history and growth of the universe up to $z=6$), the physics of inflation (constraining primordial non-Gaussianity and primordial features) and the transient radio sky (detecting one million fast radio bursts and following up SKA-discovered pulsars). We propose two array configurations composed of hexagonally close-packed 6m dish arrangements with 50% fill factor. The initial 5,000 element 'petite array' is scientifically compelling, and can act as a demonstrator and a stepping stone to the full 32,000 element 'full array'. Viewed as a 21cm intensity mapping telescope, the program has the noise equivalent of a traditional spectroscopic galaxy survey comprised of 0.6 and 2.5 billion galaxies at a comoving wavenumber of $k=0.5\,h\mathrm{Mpc}^{-1}$ spanning the redshift range $z = 0.3 - 6$ for the petite and full configurations, respectively. At redshifts beyond $z=2$, the 21cm technique is a uniquely powerful way of mapping the universe, while the low-redshift range will allow for numerous cross-correlations with existing and upcoming surveys. This program is enabled by the development of ultra-wideband radio feeds, cost-effective dish construction methods, commodity radio-frequency electronics driven by the telecommunication industry and the emergence of sufficient computing power to facilitate real-time signal processing that exploits the full potential of massive radio arrays. The project has an estimated construction cost of 55 and 330 million FY19 USD for the petite and full array configurations. Including R&D, design, operations and science analysis, the cost rises to 125 and 600 million FY19 USD, respectively.

The eROSITA X-ray telescope on SRG

Abstract: eROSITA (extended ROentgen Survey with an Imaging Telescope Array) is the primary instrument on the Spectrum-Roentgen-Gamma (SRG) mission, which was successfully launched on July 13, 2019, from the Baikonour cosmodrome. After the commissioning of the instrument and a subsequent calibration and performance verification phase, eROSITA started a survey of the entire sky on December 13, 2019. By the end of 2023, eight complete scans of the celestial sphere will have been performed, each lasting six months. At the end of this program, the eROSITA all-sky survey in the soft X-ray band (0.2--2.3\,keV) will be about 25 times more sensitive than the ROSAT All-Sky Survey, while in the hard band (2.3--8\,keV) it will provide the first ever true imaging survey of the sky. The eROSITA design driving science is the detection of large samples of galaxy clusters up to redshifts $z>1$ in order to study the large-scale structure of the universe and test cosmological models including Dark Energy. In addition, eROSITA is expected to yield a sample of a few million AGNs, including obscured objects, revolutionizing our view of the evolution of supermassive black holes. The survey will also provide new insights into a wide range of astrophysical phenomena, including X-ray binaries, active stars, and diffuse emission within the Galaxy. Results from early observations, some of which are presented here, confirm that the performance of the instrument is able to fulfil its scientific promise. With this paper, we aim to give a concise description of the instrument, its performance as measured on ground, its operation in space, and also the first results from in-orbit measurements.

The Athena X-ray Integral Field Unit (X-IFU)

Abstract: The X-ray Integral Field Unit (X-IFU) on board the Advanced Telescope for High-ENergy Astrophysics (Athena) will provide spatially resolved high-resolution X-ray spectroscopy from 0.2 to 12 keV, with 5 arc second pixels over a field of view of 5 arc minute equivalent diameter and a spectral resolution of 2.5 eV up to 7 keV. In this paper, we first review the core scientific objectives of Athena, driving the main performance parameters of the X-IFU, namely the spectral resolution, the field of view, the effective area, the count rate capabilities, the instrumental background. We also illustrate the breakthrough potential of the X-IFU for some observatory science goals. Then we briefly describe the X-IFU design as defined at the time of the mission consolidation review concluded in May 2016, and report on its predicted performance. Finally, we discuss some options to improve the instrument performance while not increasing its complexity and resource demands (e.g. count rate capability, spectral resolution). The X-IFU will be provided by an international consortium led by France, The Netherlands and Italy, with further ESA member state contributions from Belgium, Finland, Germany, Poland, Spain, Switzerland and two international partners from the United States and Japan.

List of recently commenced and planned cosmological surveys.

Measuring cross-correlations between different cosmological probes requires overlapping measurements on the sky. As shown in Fig. {{figure:3590759b-1d9e-4dbd-ae19-4c5197032942}} , the survey strategies of several operational and planned DOE-funded cosmic surveys — including optical imaging, spectroscopic, and CMB surveys — have significant overlap. While we illustrate the overlap for three specific surveys, many of the opportunities and challenges discussed here can be applied to any cross-survey analysis; we list other relevant surveys (and the corresponding acronyms used throughout the text) in Table {{table:bf02cd89-7b14-4641-98fe-fb23a83fefab}} . Given the significant overlap on the sky of future cosmic surveys, there is potential to harness the power of cross-correlations between them. However, as we discuss below, actually performing such analyses to maximize the science return from cross-correlations will require significant additional investments in simulation and analysis infrastructure, as well as mechanisms for improved cross-survey collaboration.


Very Deep Convolutional Networks for Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016

Abstract: This paper presents the method that underlies our submission to the untrimmed video classification task of ActivityNet Challenge 2016. We follow the basic pipeline of temporal segment networks and further raise the performance via a number of other techniques. Specifically, we use the latest deep model architecture, e.g., ResNet and Inception V3, and introduce new aggregation schemes (top-k and attention-weighted pooling). Additionally, we incorporate the audio as a complementary channel, extracting relevant information via a CNN applied to the spectrograms. With these techniques, we derive an ensemble of deep models, which, together, attains a high classification accuracy (mAP $93.23\%$) on the testing set and secured the first place in the challenge.

TSM: Temporal Shift Module for Efficient Video Understanding

Abstract: The explosive growth in video streaming gives rise to challenges on performing video understanding at high accuracy and low computation cost. Conventional 2D CNNs are computationally cheap but cannot capture temporal relationships; 3D CNN based methods can achieve good performance but are computationally intensive, making it expensive to deploy. In this paper, we propose a generic and effective Temporal Shift Module (TSM) that enjoys both high efficiency and high performance. Specifically, it can achieve the performance of 3D CNN but maintain 2D CNN’s complexity. TSM shifts part of the channels along the temporal dimension; thus facilitate information exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve temporal modeling at zero computation and zero parameters. We also extended TSM to online setting, which enables real-time low-latency online video recognition and video object detection. TSM is accurate and efficient: it ranks the first place on the Something-Something leaderboard upon publication; on Jetson Nano and Galaxy Note8, it achieves a low latency of 13ms and 35ms for online video recognition. The code is available at: https://github. com/mit-han-lab/temporal-shift-module.

Performance comparison on TVSeries {{cite:8820be4}}.

In Table {{table:148acfb0-ab3c-4954-b07b-12d82a5873e0}} , we summarize the results with TVSeries {{cite:8820be4}} .
For RGB input, our IEN achieves a per-frame mcAP of 81.3% and outperforms all the state-of-the-arts methods. However, our IEN that takes RGB input outperforms the RED among the state-of-the-art methods that take RGB+Flow as input. This result shows that OAD for TVSeries requires both motion and appearance information, compared to OAD for THUMOS-14 {{cite:b5b8b08}}. In TVSeries {{cite:8820be4}}, its actions that belong to different action classes occur in a similar environment.


A new autoencoder with multi-grid architecture

Learning by solving multiple view inter-prediction tasks for objects with an RNN-based network

Learning by global and local reconstruction

Learning by reconstruction and half-to-half prediction

Learning by modeling point clouds as a distribution of distributions

Learning by completing point cloud objects with a skip-attention mechanism

Learning by reconstruct point cloud objects with seed generation

Learning by completing occluded point cloud objects

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Abstract: We study the problem of 3D object generation. We propose a novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets. The benefits of our model are three-fold: first, the use of an adversarial criterion, instead of traditional heuristic criteria, enables the generator to capture object structure implicitly and to synthesize high-quality 3D objects; second, the generator establishes a mapping from a low-dimensional probabilistic space to the space of 3D objects, so that we can sample objects without a reference image or CAD models, and explore the 3D object manifold; third, the adversarial discriminator provides a powerful 3D shape descriptor which, learned without supervision, has wide applications in 3D object recognition. Experiments demonstrate that our method generates high-quality 3D objects, and our unsupervisedly learned features achieve impressive performance on 3D object recognition, comparable with those of supervised learning methods.

Learning Descriptor Networks for 3D Shape Synthesis and Analysis

Abstract: This paper proposes a 3D shape descriptor network, which is a deep convolutional energy-based model, for modeling volumetric shape patterns. The maximum likelihood training of the model follows an "analysis by synthesis" scheme and can be interpreted as a mode seeking and mode shifting process. The model can synthesize 3D shape patterns by sampling from the probability distribution via MCMC such as Langevin dynamics. The model can be used to train a 3D generator network via MCMC teaching. The conditional version of the 3D shape descriptor net can be used for 3D object recovery and 3D object super-resolution. Experiments demonstrate that the proposed model can generate realistic 3D shape patterns and can be useful for 3D shape analysis.

Multiresolution Tree Networks for 3D Point Cloud Processing

Abstract: We present multiresolution tree-structured networks to process point clouds for 3D shape understanding and generation tasks. Our network represents a 3D shape as a set of locality-preserving 1D ordered list of points at multiple resolutions. This allows efficient feed-forward processing through 1D convolutions, coarse-to-fine analysis through a multi-grid architecture, and it leads to faster convergence and small memory footprint during training. The proposed tree-structured encoders can be used to classify shapes and outperform existing point-based architectures on shape classification benchmarks, while tree-structured decoders can be used for generating point clouds directly and they outperform existing approaches for image-to-shape inference tasks learned using the ShapeNet dataset. Our model also allows unsupervised learning of point-cloud based shapes by using a variational autoencoder, leading to higher-quality generated shapes.

View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions

Abstract: In this paper we present a novel unsupervised representation learning approach for 3D shapes, which is an important research challenge as it avoids the manual effort required for collecting supervised data. Our method trains an RNN-based neural network architecture to solve multiple view inter-prediction tasks for each shape. Given several nearby views of a shape, we define view inter-prediction as the task of predicting the center view between the input views, and reconstructing the input views in a low-level feature space. The key idea of our approach is to implement the shape representation as a shape-specific global memory that is shared between all local view inter-predictions for each shape. Intuitively, this memory enables the system to aggregate information that is useful to better solve the view inter-prediction tasks for each shape, and to leverage the memory as a view-independent shape representation. Our approach obtains the best results using a combination of L_2 and adversarial losses for the view inter-prediction task. We show that VIP-GAN outperforms state-of-the-art methods in unsupervised 3D feature learning on three large scale 3D shape benchmarks.

3D Point Capsule Networks

Abstract: In this paper, we propose 3D point-capsule networks, an auto-encoder designed to process sparse 3D point clouds while preserving spatial arrangements of the input data. 3D capsule networks arise as a direct consequence of our unified formulation of the common 3D auto-encoders. The dynamic routing scheme and the peculiar 2D latent space deployed by our capsule networks bring in improvements for several common point cloud-related tasks, such as object classification, object reconstruction and part segmentation as substantiated by our extensive evaluations. Moreover, it enables new applications such as part interpolation and replacement.

L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention

Abstract: Auto-encoder is an important architecture to understand point clouds in an encoding and decoding procedure of self reconstruction. Current auto-encoder mainly focuses on the learning of global structure by global shape reconstruction, while ignoring the learning of local structures. To resolve this issue, we propose Local-to-Global auto-encoder (L2G-AE) to simultaneously learn the local and global structure of point clouds by local to global reconstruction. Specifically, L2G-AE employs an encoder to encode the geometry information of multiple scales in a local region at the same time. In addition, we introduce a novel hierarchical self-attention mechanism to highlight the important points, scales and regions at different levels in the information aggregation of the encoder. Simultaneously, L2G-AE employs a recurrent neural network (RNN) as decoder to reconstruct a sequence of scales in a local region, based on which the global point cloud is incrementally reconstructed. Our outperforming results in shape classification, retrieval and upsampling show that L2G-AE can understand point clouds better than state-of-the-art methods.

Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds From Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction

Abstract: Unsupervised feature learning for point clouds has been vital for large-scale point cloud understanding. Recent deep learning based methods depend on learning global geometry from self-reconstruction. However, these methods are still suffering from ineffective learning of local geometry, which significantly limits the discriminability of learned features. To resolve this issue, we propose MAP-VAE to enable the learning of global and local geometry by jointly leveraging global and local self-supervision. To enable effective local self-supervision, we introduce multi-angle analysis for point clouds. In a multi-angle scenario, we first split a point cloud into a front half and a back half from each angle, and then, train MAP-VAE to learn to predict a back half sequence from the corresponding front half sequence. MAP-VAE performs this half-to-half prediction using RNN to simultaneously learn each local geometry and the spatial relationship among them. In addition, MAP-VAE also learns global geometry via self-reconstruction, where we employ a variational constraint to facilitate novel shape generation. The outperforming results in four shape analysis tasks show that MAP-VAE can learn more discriminative global or local features than the state-of-the-art methods.

PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows

Abstract: As 3D point clouds become the representation of choice for multiple vision and graphics applications, the ability to synthesize or reconstruct high-resolution, high-fidelity point clouds becomes crucial. Despite the recent success of deep learning models in discriminative tasks of point clouds, generating point clouds remains challenging. This paper proposes a principled probabilistic framework to generate 3D point clouds by modeling them as a distribution of distributions. Specifically, we learn a two-level hierarchy of distributions where the first level is the distribution of shapes and the second level is the distribution of points given a shape. This formulation allows us to both sample shapes and sample an arbitrary number of points from a shape. Our generative model, named PointFlow, learns each level of the distribution with a continuous normalizing flow. The invertibility of normalizing flows enables the computation of the likelihood during training and allows us to train our model in the variational inference framework. Empirically, we demonstrate that PointFlow achieves state-of-the-art performance in point cloud generation. We additionally show that our model can faithfully reconstruct point clouds and learn useful representations in an unsupervised manner. The code is available at https://github.com/stevenygd/PointFlow.

Point Cloud Completion by Skip-attention Network with Hierarchical Folding

Abstract: Point cloud completion aims to infer the complete geometries for missing regions of 3D objects from incomplete ones. Previous methods usually predict the complete point cloud based on the global shape representation extracted from the incomplete input. However, the global representation often suffers from the information loss of structure details on local regions of incomplete point cloud. To address this problem, we propose Skip-Attention Network (SA-Net) for 3D point cloud completion. Our main contributions lie in the following two-folds. First, we propose a skip-attention mechanism to effectively exploit the local structure details of incomplete point clouds during the inference of missing parts. The skip-attention mechanism selectively conveys geometric information from the local regions of incomplete point clouds for the generation of complete ones at different resolutions, where the skip-attention reveals the completion process in an interpretable way. Second, in order to fully utilize the selected geometric information encoded by skip-attention mechanism at different resolutions, we propose a novel structure-preserving decoder with hierarchical folding for complete shape generation. The hierarchical folding preserves the structure of complete point cloud generated in upper layer by progressively detailing the local regions, using the skip-attentioned geometry at the same resolution. We conduct comprehensive experiments on ShapeNet and KITTI datasets, which demonstrate that the proposed SA-Net outperforms the state-of-the-art point cloud completion methods.

Progressive Seed Generation Auto-encoder for Unsupervised Point Cloud Learning

Abstract: With the development of 3D scanning technologies, 3D vision tasks have become a popular research area. Owing to the large amount of data acquired by sensors, unsupervised learning is essential for understanding and utilizing point clouds without an expensive annotation process. In this paper, we propose a novel framework and an effective auto-encoder architecture named "PSG-Net" for reconstruction-based learning of point clouds. Unlike existing studies that used fixed or random 2D points, our framework generates input-dependent point-wise features for the latent point set. PSG-Net uses the encoded input to produce point-wise features through the seed generation module and extracts richer features in multiple stages with gradually increasing resolution by applying the seed feature propagation module progressively. We prove the effectiveness of PSG-Net experimentally; PSG-Net shows state-of-the-art performances in point cloud reconstruction and unsupervised classification, and achieves comparable performance to counterpart methods in supervised completion.

Unsupervised Point Cloud Pre-Training via Occlusion Completion

Abstract: We describe a simple pre-training approach for point clouds. It works in three steps: 1. Mask all points occluded in a camera view; 2. Learn an encoder-decoder model to reconstruct the occluded points; 3. Use the encoder weights as initialisation for downstream point cloud tasks. We find that even when we construct a single pre-training dataset (from ModelNet40), this pre-training method improves accuracy across different datasets and encoders, on a wide range of downstream tasks. Specifically, we show that our method outperforms previous pre-training methods in object classification, and both part-based and semantic segmentation tasks. We study the pre-trained features and find that they lead to wide downstream minima, have high transformation invariance, and have activations that are highly correlated with part labels. Code and data are available at: https://github.com/hansen7/OcCo

Summary of generation-based methods of unsupervised representation learning for point clouds.

Generation-based unsupervised methods for learning point cloud representations involve the process of generating point cloud objects, which according to the pre-text tasks, can be further summarized into four subcategories including point cloud self-reconstruction (to generate point cloud objects that are the same as the input), point cloud GAN (to generate fake point cloud objects), point cloud up-sampling (to generate shape-similar but denser point clouds than the input) and point cloud completion (to predict missing parts from partial point cloud objects).
The ground truth for training these methods are point clouds themselves, which require no human annotations and can be regarded as unsupervised learning methods. A list of generation-based methods can be found in Table {{table:d4c5224b-5d20-46c6-b326-0da158164ae7}} .


Audio Wake Words Context Recognition Control Words Keyword Detection

Visual Wake Words Object Detection Gesture Recognition Object Counting Text Recognition

Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

Abstract: Describes an audio dataset of spoken words designed to help train and evaluate keyword spotting systems. Discusses why this task is an interesting challenge, and why it requires a specialized dataset that is different from conventional datasets used for automatic speech recognition of full sentences. Suggests a methodology for reproducible and comparable accuracy metrics for this task. Describes how the data was collected and verified, what it contains, previous versions and properties. Concludes by reporting baseline results of models trained on this dataset.

Visual Wake Words Dataset

Abstract: The emergence of Internet of Things (IoT) applications requires intelligence on the edge. Microcontrollers provide a low-cost compute platform to deploy intelligent IoT applications using machine learning at scale, but have extremely limited on-chip memory and compute capability. To deploy computer vision on such devices, we need tiny vision models that fit within a few hundred kilobytes of memory footprint in terms of peak usage and model size on device storage. To facilitate the development of microcontroller friendly models, we present a new dataset, Visual Wake Words, that represents a common microcontroller vision use-case of identifying whether a person is present in the image or not, and provides a realistic benchmark for tiny vision models. Within a limited memory footprint of 250 KB, several state-of-the-art mobile models achieve accuracy of 85-90% on the Visual Wake Words dataset. We anticipate the proposed dataset will advance the research on tiny vision models that can push the pareto-optimal boundary in terms of accuracy versus memory usage for microcontroller applications.

Use Cases, Models, and Datasets

Despite the general lack of maturity within the field, there are a number of well established TinyML use cases.
We categorize the application landscape of tiny ML by input type in Table {{table:af5cf52a-a271-41fd-9394-b44f6dba053b}} , which in the context of TinyML systems plays a crucial role in the use case definition.

There are a number of open-source datasets that are relevant to TinyML usecases. Table {{table:af5cf52a-a271-41fd-9394-b44f6dba053b}}  breaks them down by the type of data. Despite the availability of these datasets, the majority of deployed TinyML models are trained on much larger, proprietary datasets. The open-source datasets that are competitively large are not TinyML specific. The lack of large, TinyML focused, open-source datasets slows the progress of academic research and limits the ability of a benchmark to represent real workloads accurately.

Table {{table:af5cf52a-a271-41fd-9394-b44f6dba053b}}  lists common model types for TinyML use cases. Although neural networks (NN) are a dominant force in traditional ML, it is common to use non-NN based solutions like decision trees {{cite:0eb88c6}}, for some TinyML use cases, due to their low compute and memory requirements.


TS2Vec utilizes multi-scale contextual information at both timestamp-level and instance-level to distinguish positive and negative samples, thereby improving the generalization capability of the representation model and effectively handling time series data with missing values.

TNC learns the underlying dynamics of non-stationary signals and models the progression over time by defining a temporal neighborhood. It incorporates concepts from Positive-Unlabeled learning to account for potential bias introduced in sampling negative examples for contrastive loss.

TCL learns representations for time series that allow optimal discrimination of different time segments based on the temporal non-stationary structure captured by nonlinear independent component analysis.

BTSF applies dropout to generate diverse views for representation learning, and devises iterative bilinear temporal-spectral fusion to explicitly model pairwise cross-domain dependencies for discriminating and enriching representations in a fusion-and-squeeze manner.

CoST simulates interventions on the error variable via data augmentation and exploits prior knowledge to learn time series representations. It leverages inductive biases in the model architecture to learn disentangled seasonal and trend representations via contrastive learning.

HAR, Sleep-EDF, ECG Waveform, ETT Datasets, Weather, SAaT, WADI, SMD, SMAP, MSL

Classification, Forecasting, Anomaly Detection

TS2Vec: Towards Universal Representation of Time Series

Abstract: This paper presents TS2Vec, a universal framework for learning representations of time series in an arbitrary semantic level. Unlike existing methods, TS2Vec performs contrastive learning in a hierarchical way over augmented context views, which enables a robust contextual representation for each timestamp. Furthermore, to obtain the representation of an arbitrary sub-sequence in the time series, we can apply a simple aggregation over the representations of corresponding timestamps. We conduct extensive experiments on time series classification tasks to evaluate the quality of time series representations. As a result, TS2Vec achieves significant improvement over existing SOTAs of unsupervised time series representation on 125 UCR datasets and 29 UEA datasets. The learned timestamp-level representations also achieve superior results in time series forecasting and anomaly detection tasks. A linear regression trained on top of the learned representations outperforms previous SOTAs of time series forecasting. Furthermore, we present a simple way to apply the learned representations for unsupervised anomaly detection, which establishes SOTA results in the literature. The source code is publicly available at https://github.com/yuezhihan/ts2vec.

Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding

Abstract: Time series are often complex and rich in information but sparsely labeled and therefore challenging to model. In this paper, we propose a self-supervised framework for learning generalizable representations for non-stationary time series. Our approach, called Temporal Neighborhood Coding (TNC), takes advantage of the local smoothness of a signal's generative process to define neighborhoods in time with stationary properties. Using a debiased contrastive objective, our framework learns time series representations by ensuring that in the encoding space, the distribution of signals from within a neighborhood is distinguishable from the distribution of non-neighboring signals. Our motivation stems from the medical field, where the ability to model the dynamic nature of time series data is especially valuable for identifying, tracking, and predicting the underlying patients' latent states in settings where labeling data is practically impossible. We compare our method to recently developed unsupervised representation learning approaches and demonstrate superior performance on clustering and classification tasks for multiple datasets.

Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA

Abstract: Nonlinear independent component analysis (ICA) provides an appealing framework for unsupervised feature learning, but the models proposed so far are not identifiable. Here, we first propose a new intuitive principle of unsupervised deep learning from time series which uses the nonstationary structure of the data. Our learning principle, time-contrastive learning (TCL), finds a representation which allows optimal discrimination of time segments (windows). Surprisingly, we show how TCL can be related to a nonlinear ICA model, when ICA is redefined to include temporal nonstationarities. In particular, we show that TCL combined with linear ICA estimates the nonlinear ICA model up to point-wise transformations of the sources, and this solution is unique --- thus providing the first identifiability result for nonlinear ICA which is rigorous, constructive, as well as very general.

Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion

Abstract: Unsupervised/self-supervised time series representation learning is a challenging problem because of its complex dynamics and sparse annotations. Existing works mainly adopt the framework of contrastive learning with the time-based augmentation techniques to sample positives and negatives for contrastive training. Nevertheless, they mostly use segment-level augmentation derived from time slicing, which may bring about sampling bias and incorrect optimization with false negatives due to the loss of global context. Besides, they all pay no attention to incorporate the spectral information in feature representation. In this paper, we propose a unified framework, namely Bilinear Temporal-Spectral Fusion (BTSF). Specifically, we firstly utilize the instance-level augmentation with a simple dropout on the entire time series for maximally capturing long-term dependencies. We devise a novel iterative bilinear temporal-spectral fusion to explicitly encode the affinities of abundant time-frequency pairs, and iteratively refines representations in a fusion-and-squeeze manner with Spectrum-to-Time (S2T) and Time-to-Spectrum (T2S) Aggregation modules. We firstly conducts downstream evaluations on three major tasks for time series including classification, forecasting and anomaly detection. Experimental results shows that our BTSF consistently significantly outperforms the state-of-the-art methods.

CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting

Abstract: Deep learning has been actively studied for time series forecasting, and the mainstream paradigm is based on the end-to-end training of neural network architectures, ranging from classical LSTM/RNNs to more recent TCNs and Transformers. Motivated by the recent success of representation learning in computer vision and natural language processing, we argue that a more promising paradigm for time series forecasting, is to first learn disentangled feature representations, followed by a simple regression fine-tuning step -- we justify such a paradigm from a causal perspective. Following this principle, we propose a new time series representation learning framework for time series forecasting named CoST, which applies contrastive learning methods to learn disentangled seasonal-trend representations. CoST comprises both time domain and frequency domain contrastive losses to learn discriminative trend and seasonal representations, respectively. Extensive experiments on real-world datasets show that CoST consistently outperforms the state-of-the-art methods by a considerable margin, achieving a 21.3% improvement in MSE on multivariate benchmarks. It is also robust to various choices of backbone encoders, as well as downstream regressors. Code is available at https://github.com/salesforce/CoST.

Summary of temporal-level contrastive methods for time series representation learning. DA indicates whether the model utilizes data augmentations. ACC refers to accuracy, AUPRC refers to the area under the precision-recall curve, DBI refers to Davies-Bouldin Index, F1 refers to F1 score, MF1 refers to macro-averaged F1 score, MSE refers to mean square error, and MAE refers to mean absolute error.

Contrastive methods learn meaningful representations from time series by optimizing self-discrimination tasks. Instead of directly modeling the complex raw data, they employ pretext tasks that leverage the underlying similarity between samples, which eliminates the need for reconstructing the complete input and allows for the discovery of contextualized underlying factors of variations {{cite:51941f9}}. Contrastive methods typically generate augmented views of the raw data through various transformations and then learn representations by contrasting positive samples against negative samples {{cite:274251f}}. Exploring negative samples in a completely unsupervised manner can be challenging for contrastive learning, as it often encounters hard negative samples with features that are highly similar to anchors but have different labels. To address this, researchers have explored various techniques to introduce more negatives, such as increasing the batch size {{cite:b7f4787}} or exploiting external data structures {{cite:8f338a9}}, {{cite:3bec721}}. There are also studies that remove the need of using negative samples and instead use other strategies such as utilizing a predictor {{cite:c57d039}}, a stop-gradient operation {{cite:5565e4d}}, or clustering {{cite:c850344}} to learn the mapping. Within the realm of computer vision, contrastive methods are commonly categorized into instance-level and prototype-level methods, with each approach focusing on different levels of contrast. Additionally, there are specific methods tailored for time series data, which consider the temporal-level contrast. We analyze the similarities and differences between these representative contrastive models in terms of their main contributions, backbone architectures, data augmentations (DA), datasets, evaluation strategies, and performance metrics. Contrastive learning enables the discovery of informative representations in time series data without relying on explicit reconstruction tasks. This analysis provides insights into the key characteristics and approaches of representative contrastive models in the field of representation learning for time series data. The concise overview of instance-level, prototype-level, and temporal-level methods can be found in Table {{table:6c7eb62e-ed78-4d78-897c-524de22a05fd}} , Table {{table:d8fc4852-a4da-4cbb-82d8-8484378451f4}} , and Table {{table:db6f7e97-94ad-457e-822e-6b784da1befd}} , respectively. The following sections provide more in-depth details about these methods.

Temporal-level Contrastive Learning Models

Although contrastive learning methods have demonstrated remarkable success in the field of computer vision, their application to time series data often ignores the intricate characteristics that are inherent to such data. Acknowledging this limitation, researchers have recently turned their attention towards investigating the influence of timestamps, and have begun developing specifically tailored strategies to address the challenge posed by time series data. Instance-level contrastive learning models are able to learn general representations that capture the overall characteristics of the entire time series. These representations provide a holistic view of data and can be useful for tasks that require a global understanding of the sequence. Temporal-level contrastive learning models instead focus on capturing scale-invariant representations at each individual timestamp. By considering both instance-level and temporal-level representation learning strategies, researchers aim to enhance the capability of contrastive learning methods in capturing the complexities inherent in time series data. Existing temporal-level contrastive learning models either consider the temporal dependencies by leveraging temporal contrasting modules {{cite:ef3ec4d}}, {{cite:93c2f6b}}, {{cite:1f13dcd}}, or focus on capturing multi-scale contextual information across different granularities {{cite:08a1147}}, {{cite:db8fe16}}, {{cite:6efe386}}, {{cite:522180d}}. Approaches such as T-Loss {{cite:93c2f6b}} and TNC {{cite:1f13dcd}} utilize the information from the neighborhood to construct positive and negative samples for contrastive learning. TS-TCC {{cite:ef3ec4d}} instead designs time-series-specific augmented views for pair constructions and performs cross-view temporal contrasting to learn representations. There also exist methods that explicitly emphasize the timestamp-level {{cite:08a1147}} or segment-level {{cite:db8fe16}}, {{cite:6efe386}}, {{cite:522180d}} representations in addition to the entire time-series level. These models are designed to capture fine-grained temporal patterns and variations, enabling them to extract more detailed and localized information from time series. However, they may ignore the higher-level semantic information which is involved in the entire set of time series. Incorporating potential semantic information such as class labels can help eliminate fake negative samples of the same class, thereby reducing noise in contrastive learning. Table {{table:db6f7e97-94ad-457e-822e-6b784da1befd}}  provides a succinct summary of temporal-level contrastive methods for time series representation learning. Further elaboration on these methods is explicated below.

A comparative analysis of using different backbones

Table {{table:44b9bdb3-230c-4ef7-9cd1-98b788cb1f83}}  presents a comparative analysis of employing different backbones in the SimCLR model for unsupervised time series representation learning. Typically, we find that most models summarized in Table {{table:6c7eb62e-ed78-4d78-897c-524de22a05fd}} , Table {{table:d8fc4852-a4da-4cbb-82d8-8484378451f4}}  and Table {{table:db6f7e97-94ad-457e-822e-6b784da1befd}}  utilize ResNets {{cite:a30ba3d}} as the backbone and achieve satisfactory performance. Therefore, we compare the results of using ResNet-18 and ResNet-50 as backbones. Furthermore, we evaluate InceptionTime {{cite:653c9a1}}, which is specifically designed for time series data by incorporating the temporal dependencies. As shown in Table {{table:44b9bdb3-230c-4ef7-9cd1-98b788cb1f83}} , InceptionTime outperforms the general-purpose ResNets in most datasets. This observation highlights the capability of InceptionTime to identify both local and global shape patterns (i.e., low-level and high-level features) in time series data. InceptionTime proves to be exceptionally fitting for time series data due to its utilization of 1D convolutions with multiple kernel sizes in parallel. This approach facilitates efficient feature extraction on time series data across various scales, making it well-suited to capture diverse granularities of semantics in time series data. In contrast, ResNets primarily rely on 2D convolutions and are optimized for image data, which may not yield the same level of effectiveness when applied to time series data analysis.


Explaining Relationships Between Scientific Documents

Abstract: We address the task of explaining relationships between two scientific documents using natural language text. This task requires modeling the complex content of long technical documents, deducing a relationship between these documents, and expressing the details of that relationship in text. In addition to the theoretical interest of this task, successful solutions can help improve researcher efficiency in search and review. In this paper we establish a dataset of 622K examples from 154K documents. We pretrain a large language model to serve as the foundation for autoregressive approaches to the task. We explore the impact of taking different views on the two documents, including the use of dense representations extracted with scientific IE systems. We provide extensive automatic and human evaluations which show the promise of such models, but make clear challenges for future work.

Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation

Abstract: Conventional neural networks show a powerful framework for background subtraction in video acquired by static cameras. Indeed, the well-known SOBS method and its variants based on neural networks were the leader methods on the largescale CDnet 2012 dataset during a long time. Recently, convolutional neural networks which belong to deep learning methods were employed with success for background initialization, foreground detection and deep learned features. Currently, the top current background subtraction methods in CDnet 2014 are based on deep neural networks with a large gap of performance in comparison on the conventional unsupervised approaches based on multi-features or multi-cues strategies. Furthermore, a huge amount of papers was published since 2016 when Braham and Van Droogenbroeck published their first work on CNN applied to background subtraction providing a regular gain of performance. In this context, we provide the first review of deep neural network concepts in background subtraction for novices and experts in order to analyze this success and to provide further directions. For this, we first surveyed the methods used background initialization, background subtraction and deep learned features. Then, we discuss the adequacy of deep neural networks for background subtraction. Finally, experimental results are presented on the CDnet 2014 dataset.

A summary of the datasets of the prior works on related work generation.

Overview of Datasets Used in Related Work Generation

Table {{table:a47d897a-5f2b-48fd-9814-a2b14174e878}}  summarizes the datasets of the surveyed studies collect. Because of the professionality of academic papers, reading papers from unfamiliar domains is challenging. Therefore, prior works mostly focus on the computational linguistic domain that NLP researchers are naturally familiar with.


Key-Value Memory Networks for Directly Reading Documents

Abstract: Directly reading documents and being able to answer questions from them is an unsolved challenge. To avoid its inherent difficulty, question answering (QA) has been directed towards using Knowledge Bases (KBs) instead, which has proven effective. Unfortunately KBs often suffer from being too restrictive, as the schema cannot support certain types of answers, and too sparse, e.g. Wikipedia contains much more information than Freebase. In this work we introduce a new method, Key-Value Memory Networks, that makes reading documents more viable by utilizing different encodings in the addressing and output stages of the memory read operation. To compare using KBs, information extraction or Wikipedia documents directly in a single framework we construct an analysis tool, WikiMovies, a QA dataset that contains raw text alongside a preprocessed KB, in the domain of movies. Our method reduces the gap between all three settings. It also achieves state-of-the-art results on the existing WikiQA benchmark.

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Abstract: We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com

Quasar: Datasets for Question Answering by Search and Reading

Abstract: We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S dataset consists of 37000 cloze-style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. The posts and comments on the website serve as the background corpus for answering the cloze questions. The Quasar-T dataset consists of 43000 open-domain trivia questions and their answers obtained from various internet sources. ClueWeb09 serves as the background corpus for extracting these answers. We pose these datasets as a challenge for two related subtasks of factoid Question Answering: (1) searching for relevant pieces of text that include the correct answer to a query, and (2) reading the retrieved text to answer the query. We also describe a retrieval system for extracting relevant sentences and documents from the corpus given a query, and include these in the release for researchers wishing to only focus on (2). We evaluate several baselines on both datasets, ranging from simple heuristics to powerful neural models, and show that these lag behind human performance by 16.4% and 32.1% for Quasar-S and -T respectively. The datasets are available at https://github.com/bdhingra/quasar .

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Abstract: We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a feature-based classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that TriviaQA is a challenging testbed that is worth significant future study. Data and code available at -- http://nlp.cs.washington.edu/triviaqa/

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Abstract: Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

ELI5: Long Form Question Answering

Abstract: We introduce the first large-scale corpus for long-form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum ``Explain Like I'm Five'' (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline. However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.

Open-domain question answering benchmarks.

Theoretically, because of the end-to-end nature of ODQA, any existing QA dataset that contains objective question-answer pairs can be curated to serve in this scenario by pairing it with a large external knowledge source that can be used to answer its questions, e.g., Wikipedia.
Hence, besides those datasets that were originally designed for ODQA (e.g., CuratedTREC{{cite:338268e}}, Quasar-T{{cite:14da6a3}}), many famous large-scale MRC datasets(e.g., SQuAD{{cite:a4e676c}}, TRIVIA QA{{cite:af31daa}}, Natural Questions{{cite:d3ccfa7}}, etc.) were curated to serve as benchmarks for ODQA systems(e.g., DrQA{{cite:16b9a1d}}, ORQA{{cite:44fbad0}}, etc.).
Next, we will introduce the most widely-used benchmarks in modern ODQA systems. The information on these datasets is summarized in Table {{table:f2fafd4c-406e-428c-9379-e272944edbd6}} .


ParamILS: An Automatic Algorithm Configuration Framework

Abstract: The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms. We describe an automatic framework for this algorithm configuration problem. More formally, we provide methods for optimizing a target algorithm's performance on a given class of problem instances by varying a set of ordinal and/or categorical parameters. We review a family of local-search-based algorithm configuration procedures and present novel techniques for accelerating them by adaptively limiting the time spent for evaluating individual configurations. We describe the results of a comprehensive experimental evaluation of our methods, based on the configuration of prominent complete and incomplete algorithms for SAT. We also present what is, to our knowledge, the first published work on automatically configuring the CPLEX mixed integer programming solver. All the algorithms we considered had default parameter settings that were manually identified with considerable effort. Nevertheless, using our automated algorithm configuration procedures, we achieved substantial and consistent performance improvements.

MATE: A Model-based Algorithm Tuning Engine

Abstract: In this paper, we introduce a Model-based Algorithm Turning Engine, namely MATE, where the parameters of an algorithm are represented as expressions of the features of a target optimisation problem. In contrast to most static (feature-independent) algorithm tuning engines such as irace and SPOT, our approach aims to derive the best parameter configuration of a given algorithm for a specific problem, exploiting the relationships between the algorithm parameters and the features of the problem. We formulate the problem of finding the relationships between the parameters and the problem features as a symbolic regression problem and we use genetic programming to extract these expressions. For the evaluation, we apply our approach to configuration of the (1+1) EA and RLS algorithms for the OneMax, LeadingOnes, BinValue and Jump optimisation problems, where the theoretically optimal algorithm parameters to the problems are available as functions of the features of the problems. Our study shows that the found relationships typically comply with known theoretical results, thus demonstrating a new opportunity to consider model-based parameter tuning as an effective alternative to the static algorithm tuning engines.

Summary of the state-of-the-art related works

Table {{table:02f746e4-f74a-46d4-8618-a634153c3037}}  presents a summary for some state-of-the-art methods including the approach proposed in this paper.
The term feature-independent means that the corresponding approach does not consider instance features.
Model-based approaches use a trained model (e.g. machine learning, regression, etc.) to design parameters configuration.
Model-free approaches generally rely on an experimental design methodology or optimisation method to find parameter settings of an algorithm that optimise a cost metric on a given instance set.