Aggregating neighbors using the sum operator

Performing message passing between subgraph structures instead of the node level

Considering higher-order message passing

Using a ring of matrices in addition and multiplication

Augmenting the model with matrix multiplication

Designing the convolution in spectral domain and masking it with a large receptive field

Adding random features to GIN {{cite:a74f4c0}}

Using separate channels to update the node and positional features

Encoding a rooted subgraph for each node instead of a rooted subtree

Inductively injecting node identities either using coloring mechanism or an augmented node feature

Using colors to distinguish similar node attributes

How Powerful are Graph Neural Networks?

Abstract: Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. Many GNN variants have been proposed and have achieved state-of-the-art results on both node and graph classification tasks. However, despite GNNs revolutionizing graph representation learning, there is limited understanding of their representational properties and limitations. Here, we present a theoretical framework for analyzing the expressive power of GNNs to capture different graph structures. Our results characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures. We then develop a simple architecture that is provably the most expressive among the class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism test. We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance.

Relational Pooling for Graph Representations

Abstract: This work generalizes graph neural networks (GNNs) beyond those based on the Weisfeiler-Lehman (WL) algorithm, graph Laplacians, and diffusions. Our approach, denoted Relational Pooling (RP), draws from the theory of finite partial exchangeability to provide a framework with maximal representation power for graphs. RP can work with existing graph representation models and, somewhat counterintuitively, can make them even more powerful than the original WL isomorphism test. Additionally, RP allows architectures like Recurrent Neural Networks and Convolutional Neural Networks to be used in a theoretically sound approach for graph classification. We demonstrate improved performance of RP-based graph representations over state-of-the-art methods on a number of tasks.

Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks

Abstract: In recent years, graph neural networks (GNNs) have emerged as a powerful neural architecture to learn vector representations of nodes and graphs in a supervised, end-to-end fashion. Up to now, GNNs have only been evaluated empirically—showing promising results. The following work investigates GNNs from a theoretical point of view and relates them to the 1-dimensional Weisfeiler-Leman graph isomorphism heuristic (1-WL). We show that GNNs have the same expressiveness as the 1-WL in terms of distinguishing non-isomorphic (sub-)graphs. Hence, both algorithms also have the same shortcomings. Based on this, we propose a generalization of GNNs, so-called k-dimensional GNNs (k-GNNs), which can take higher-order graph structures at multiple scales into account. These higher-order structures play an essential role in the characterization of social networks and molecule graphs. Our experimental evaluation confirms our theoretical findings as well as confirms that higher-order information is useful in the task of graph classification and regression.

Provably Powerful Graph Networks

Abstract: Recently, the Weisfeiler-Lehman (WL) graph isomorphism test was used to measure the expressive power of graph neural networks (GNN). It was shown that the popular message passing GNN cannot distinguish between graphs that are indistinguishable by the 1-WL test (Morris et al. 2018; Xu et al. 2019). Unfortunately, many simple instances of graphs are indistinguishable by the 1-WL test. In search for more expressive graph learning models we build upon the recent k-order invariant and equivariant graph neural networks (Maron et al. 2019a,b) and present two results: First, we show that such k-order networks can distinguish between non-isomorphic graphs as good as the k-WL tests, which are provably stronger than the 1-WL test for k>2. This makes these models strictly stronger than message passing models. Unfortunately, the higher expressiveness of these models comes with a computational cost of processing high order tensors. Second, setting our goal at building a provably stronger, simple and scalable model we show that a reduced 2-order network containing just scaled identity operator, augmented with a single quadratic operation (matrix multiplication) has a provable 3-WL expressive power. Differently put, we suggest a simple model that interleaves applications of standard Multilayer-Perceptron (MLP) applied to the feature dimension and matrix multiplication. We validate this model by presenting state of the art results on popular graph classification and regression tasks. To the best of our knowledge, this is the first practical invariant/equivariant model with guaranteed 3-WL expressiveness, strictly stronger than message passing models.

On the equivalence between graph isomorphism testing and function approximation with GNNs

Abstract:
Graph neural networks (GNNs) have achieved lots of success on graph-structured data. In the light of this, there has been increasing interest in studying their representation power. One line of work focuses on the universal approximation of permutation-invariant functions by certain classes of GNNs, and another demonstrates the limitation of GNNs via graph isomorphism tests.
Our work connects these two perspectives and proves their equivalence. We further develop a framework of the representation power of GNNs with the language of sigma-algebra, which incorporates both viewpoints. Using this framework, we compare the expressive power of different classes of GNNs as well as other methods on graphs. In particular, we prove that order-2 Graph G-invariant networks fail to distinguish non-isomorphic regular graphs with the same degree. We then extend them to a new architecture, Ring-GNNs, which succeeds on distinguishing these graphs and provides improvements on real-world social network datasets.

Expressive Power of Invariant and Equivariant Graph Neural Networks

Abstract: Various classes of Graph Neural Networks (GNN) have been proposed and shown to be successful in a wide range of applications with graph structured data. In this paper, we propose a theoretical framework able to compare the expressive power of these GNN architectures. The current universality theorems only apply to intractable classes of GNNs. Here, we prove the first approximation guarantees for practical GNNs, paving the way for a better understanding of their generalization. Our theoretical results are proved for invariant GNNs computing a graph embedding (permutation of the nodes of the input graph does not affect the output) and equivariant GNNs computing an embedding of the nodes (permutation of the input permutes the output). We show that Folklore Graph Neural Networks (FGNN), which are tensor based GNNs augmented with matrix multiplication are the most expressive architectures proposed so far for a given tensor order. We illustrate our results on the Quadratic Assignment Problem (a NP-Hard combinatorial problem) by showing that FGNNs are able to learn how to solve the problem, leading to much better average performances than existing algorithms (based on spectral, SDP or other GNNs architectures). On a practical side, we also implement masked tensors to handle batches of graphs of varying sizes.

Breaking the Limits of Message Passing Graph Neural Networks

Abstract: Since the Message Passing (Graph) Neural Networks (MPNNs) have a linear complexity with respect to the number of nodes when applied to sparse graphs, they have been widely implemented and still raise a lot of interest even though their theoretical expressive power is limited to the first order Weisfeiler-Lehman test (1-WL). In this paper, we show that if the graph convolution supports are designed in spectral-domain by a non-linear custom function of eigenvalues and masked with an arbitrary large receptive field, the MPNN is theoretically more powerful than the 1-WL test and experimentally as powerful as a 3-WL existing models, while remaining spatially localized. Moreover, by designing custom filter functions, outputs can have various frequency components that allow the convolution process to learn different relationships between a given input graph signal and its associated properties. So far, the best 3-WL equivalent graph neural networks have a computational complexity in $\mathcal{O}(n^3)$ with memory usage in $\mathcal{O}(n^2)$, consider non-local update mechanism and do not provide the spectral richness of output profile. The proposed method overcomes all these aforementioned problems and reaches state-of-the-art results in many downstream tasks.

Graph Neural Networks with Local Graph Parameters

Abstract: Various recent proposals increase the distinguishing power of Graph Neural Networks GNNs by propagating features between $k$-tuples of vertices. The distinguishing power of these "higher-order'' GNNs is known to be bounded by the $k$-dimensional Weisfeiler-Leman (WL) test, yet their $\mathcal O(n^k)$ memory requirements limit their applicability. Other proposals infuse GNNs with local higher-order graph structural information from the start, hereby inheriting the desirable $\mathcal O(n)$ memory requirement from GNNs at the cost of a one-time, possibly non-linear, preprocessing step. We propose local graph parameter enabled GNNs as a framework for studying the latter kind of approaches and precisely characterize their distinguishing power, in terms of a variant of the WL test, and in terms of the graph structural properties that they can take into account. Local graph parameters can be added to any GNN architecture, and are cheap to compute. In terms of expressive power, our proposal lies in the middle of GNNs and their higher-order counterparts. Further, we propose several techniques to aide in choosing the right local graph parameters. Our results connect GNNs with deep results in finite model theory and finite variable logics. Our experimental evaluation shows that adding local graph parameters often has a positive effect for a variety of GNNs, datasets and graph learning tasks.

Random Features Strengthen Graph Neural Networks

Abstract: Graph neural networks (GNNs) are powerful machine learning models for various graph learning tasks. Recently, the limitations of the expressive power of various GNN models have been revealed. For example, GNNs cannot distinguish some non-isomorphic graphs and they cannot learn efficient graph algorithms, and several GNN models have been proposed to overcome these limitations. In this paper, we demonstrate that GNNs become powerful just by adding a random feature to each node. We prove that the random features enable GNNs to learn almost optimal polynomial-time approximation algorithms for the minimum dominating set problem and maximum matching problem in terms of the approximation ratio. The main advantage of our method is that it can be combined with off-the-shelf GNN models with slight modifications. Through experiments, we show that the addition of random features enables GNNs to solve various problems that normal GNNs, including GCNs and GINs, cannot solve.

DropGNN: Random Dropouts Increase the Expressiveness of Graph Neural Networks

Abstract: This paper studies Dropout Graph Neural Networks (DropGNNs), a new approach that aims to overcome the limitations of standard GNN frameworks. In DropGNNs, we execute multiple runs of a GNN on the input graph, with some of the nodes randomly and independently dropped in each of these runs. Then, we combine the results of these runs to obtain the final result. We prove that DropGNNs can distinguish various graph neighborhoods that cannot be separated by message passing GNNs. We derive theoretical bounds for the number of runs required to ensure a reliable distribution of dropouts, and we prove several properties regarding the expressive capabilities and limits of DropGNNs. We experimentally validate our theoretical findings on expressiveness. Furthermore, we show that DropGNNs perform competitively on established GNN benchmarks.

Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks

Abstract: Graph neural networks (GNN) have shown great advantages in many graph-based learning tasks but often fail to predict accurately for a task-based on sets of nodes such as link/motif prediction and so on. Many works have recently proposed to address this problem by using random node features or node distance features. However, they suffer from either slow convergence, inaccurate prediction, or high complexity. In this work, we revisit GNNs that allow using positional features of nodes given by positional encoding (PE) techniques such as Laplacian Eigenmap, Deepwalk, etc. GNNs with PE often get criticized because they are not generalizable to unseen graphs (inductive) or stable. Here, we study these issues in a principled way and propose a provable solution, a class of GNN layers termed PEG with rigorous mathematical analysis. PEG uses separate channels to update the original node features and positional features. PEG imposes permutation equivariance w.r.t. the original node features and imposes $O(p)$ (orthogonal group) equivariance w.r.t. the positional features simultaneously, where $p$ is the dimension of used positional features. Extensive link prediction experiments over 8 real-world networks demonstrate the advantages of PEG in generalization and scalability.

Nested Graph Neural Networks

Abstract: Graph neural network (GNN)'s success in graph classification is closely related to the Weisfeiler-Lehman (1-WL) algorithm. By iteratively aggregating neighboring node features to a center node, both 1-WL and GNN obtain a node representation that encodes a rooted subtree around the center node. These rooted subtree representations are then pooled into a single representation to represent the whole graph. However, rooted subtrees are of limited expressiveness to represent a non-tree graph. To address it, we propose Nested Graph Neural Networks (NGNNs). NGNN represents a graph with rooted subgraphs instead of rooted subtrees, so that two graphs sharing many identical subgraphs (rather than subtrees) tend to have similar representations. The key is to make each node representation encode a subgraph around it more than a subtree. To achieve this, NGNN extracts a local subgraph around each node and applies a base GNN to each subgraph to learn a subgraph representation. The whole-graph representation is then obtained by pooling these subgraph representations. We provide a rigorous theoretical analysis showing that NGNN is strictly more powerful than 1-WL. In particular, we proved that NGNN can discriminate almost all r-regular graphs, where 1-WL always fails. Moreover, unlike other more powerful GNNs, NGNN only introduces a constant-factor higher time complexity than standard GNNs. NGNN is a plug-and-play framework that can be combined with various base GNNs. We test NGNN with different base GNNs on several benchmark datasets. NGNN uniformly improves their performance and shows highly competitive performance on all datasets.

Identity-aware Graph Neural Networks

Abstract: Message passing Graph Neural Networks (GNNs) provide a powerful modeling framework for relational data. However, the expressive power of existing GNNs is upper-bounded by the 1-Weisfeiler-Lehman (1-WL) graph isomorphism test, which means GNNs that are not able to predict node clustering coefficients and shortest path distances, and cannot differentiate between different d-regular graphs. Here we develop a class of message passing GNNs, named Identity-aware Graph Neural Networks (ID-GNNs), with greater expressive power than the 1-WL test. ID-GNN offers a minimal but powerful solution to limitations of existing GNNs. ID-GNN extends existing GNN architectures by inductively considering nodes’ identities during message passing. To embed a given node, ID-GNN first extracts the ego network centered at the node, then conducts rounds of heterogeneous message passing, where different sets of parameters are applied to the center node than to other surrounding nodes in the ego network. We further propose a simplified but faster version of ID-GNN that injects node identity information as augmented node features. Alto- gether, both versions of ID-GNN represent general extensions of message passing GNNs, where experiments show that transforming existing GNNs to ID-GNNs yields on average 40% accuracy improvement on challenging node, edge, and graph property prediction tasks; 3% accuracy improvement on node and graph classification benchmarks; and 15% ROC AUC improvement on real-world link prediction tasks. Additionally, ID-GNNs demonstrate improved or comparable performance over other task-specific graph networks.

Coloring graph neural networks for node disambiguation

Abstract: In this paper, we show that a simple coloring scheme can improve, both theoretically and empirically, the expressive power of Message Passing Neural Networks(MPNNs). More specifically, we introduce a graph neural network called Colored Local Iterative Procedure (CLIP) that uses colors to disambiguate identical node attributes, and show that this representation is a universal approximator of continuous functions on graphs with node attributes. Our method relies on separability , a key topological characteristic that allows to extend well-chosen neural networks into universal representations. Finally, we show experimentally that CLIP is capable of capturing structural characteristics that traditional MPNNs fail to distinguish,while being state-of-the-art on benchmark graph classification datasets.

A summary of major solutions proposed to increase the expressive power of GNNs

where {{formula:8defe37c-11d4-4255-8d91-a0f880b3861d}}  can be a fixed or learnable parameter. GIN uses the sum aggregator as its expressive power is higher than the mean or max operators. For instance, assume we have two nodes {{formula:7cbe7160-6e00-4922-bf88-0225f0cc3010}}  and {{formula:488bea67-30c7-4bed-a042-007d714c5268}} . If {{formula:9f759103-1ca9-49ed-98a3-146863bd6d86}}  has two equal neighbors and {{formula:614c3832-91b3-47e1-b569-ae8291e07829}}  has three equal neighbors. The mean aggregator generates the same embedding for two nodes. The same applies to the max aggregator. However, the sum operator distinguishes different graph structures and generates different embeddings. It is also theoretically proven that GIN's expressive power is equal to the WL test.
Identity-aware Graph Neural Networks (ID-GNN) {{cite:7f3f40b}} proposes a coloring mechanism as a solution for increasing the expressive power of GNNs. This model has two variants. The first variant has two components: inductive identity coloring and heterogeneous message passing. For computing the embedding for each node {{formula:f5ac6f88-8853-4fbf-acd9-aa781106274b}} , a k-hop ego networks of node {{formula:0d0af1f3-07c0-402e-b8cf-d77c4d25a267}}  is extracted and the center node of the ego network is colored. Then, for each node in the ego network of node {{formula:ec39c029-4044-48d0-97db-dbf899f60eda}} , the embedding is computed using a different message passing component for each node based on color. This paper proposes a fast variant of ID-GNN that augments the node features instead of coloring nodes by injecting identity information such as cycle counts for cycles that start and end in the node {{formula:8ad4c651-febb-4762-95f9-237fbe5af203}} .
Nested Graph Neural Net (NGNN) {{cite:e4d3898}} suggests to encode a rooted subgraph instead of rooted subtree in common GNNs to generate node representations. It argues that rooted subtrees have limited expressiveness to represent non-tree graphs. In {{cite:ae1f82b}}, {{cite:6dff544}}, distance-based features are added to each node to increase the expressive power of GNNs, where distance vectors for each node are computed with respect to each center node. Table {{table:2c938b71-31ff-4c43-8e34-88d636be4479}}  summarizes the major studies related to increasing the expressive power of GNNs.


single-stream Siamese Nets {{cite:a23ef5a}}

Learning feed-forward one-shot learners

Abstract: One-shot learning is usually tackled by using generative models or discriminative embeddings. Discriminative methods based on deep learning, which are very effective in other learning scenarios, are ill-suited for one-shot learning as they need large amounts of training data. In this paper, we propose a method to learn the parameters of a deep model in one shot. We construct the learner as a second deep network, called a learnet, which predicts the parameters of a pupil network from a single exemplar. In this manner we obtain an efficient feed-forward one-shot learner, trained end-to-end by minimizing a one-shot classification objective in a learning to learn formulation. In order to make the construction feasible, we propose a number of factorizations of the parameters of the pupil network. We demonstrate encouraging results by learning characters from single exemplars in Omniglot, and by tracking visual objects from a single initial exemplar in the Visual Object Tracking benchmark.

Few-Shot Image Recognition by Predicting Parameters from Activations

Abstract: In this paper, we are interested in the few-shot learning problem. In particular, we focus on a challenging scenario where the number of categories is large and the number of examples per novel category is very limited, e.g. 1, 2, or 3. Motivated by the close relationship between the parameters and the activations in a neural network associated with the same category, we propose a novel method that can adapt a pre-trained neural network to novel categories by directly predicting the parameters from the activations. Zero training is required in adaptation to novel categories, and fast inference is realized by a single forward pass. We evaluate our method by doing few-shot image recognition on the ImageNet dataset, which achieves the state-of-the-art classification accuracy on novel categories by a significant margin while keeping comparable performance on the large-scale categories. We also test our method on the MiniImageNet dataset and it strongly outperforms the previous state-of-the-art methods.

Meta-learning autoencoders for few-shot prediction

Abstract: Compared to humans, machine learning models generally require significantly more training examples and fail to extrapolate from experience to solve previously unseen challenges. To help close this performance gap, we augment single-task neural networks with a meta-recognition model which learns a succinct model code via its autoencoder structure, using just a few informative examples. The model code is then employed by a meta-generative model to construct parameters for the task-specific model. We demonstrate that for previously unseen tasks, without additional training, this Meta-Learning Autoencoder (MeLA) framework can build models that closely match the true underlying models, with loss significantly lower than given by fine-tuned baseline networks, and performance that compares favorably with state-of-the-art meta-learning algorithms. MeLA also adds the ability to identify influential training examples and predict which additional data will be most valuable to acquire to improve model prediction.

Generating Classification Weights With GNN Denoising Autoencoders for Few-Shot Learning

Abstract: Given an initial recognition model already trained on a set of base classes, the goal of this work is to develop a meta-model for few-shot learning. The meta-model, given as input some novel classes with few training examples per class, must properly adapt the existing recognition model into a new model that can correctly classify in a unified way both the novel and the base classes. To accomplish this goal it must learn to output the appropriate classification weight vectors for those two types of classes. To build our meta-model we make use of two main innovations: we propose the use of a Denoising Autoencoder network (DAE) that (during training) takes as input a set of classification weights corrupted with Gaussian noise and learns to reconstruct the target-discriminative classification weights. In this case, the injected noise on the classification weights serves the role of regularizing the weight generating meta-model. Furthermore, in order to capture the co-dependencies between different classes in a given task instance of our meta-model, we propose to implement the DAE model as a Graph Neural Network (GNN). In order to verify the efficacy of our approach, we extensively evaluate it on ImageNet based few-shot benchmarks and we report state-of-the-art results.

Meta-Learning Probabilistic Inference for Prediction

Abstract: This paper introduces a new framework for data efficient and versatile learning. Specifically: 1) We develop ML-PIP, a general framework for Meta-Learning approximate Probabilistic Inference for Prediction. ML-PIP extends existing probabilistic interpretations of meta-learning to cover a broad class of methods. 2) We introduce VERSA, an instance of the framework employing a flexible and versatile amortization network that takes few-shot learning datasets as inputs, with arbitrary numbers of shots, and outputs a distribution over task-specific parameters in a single forward pass. VERSA substitutes optimization at test time with forward passes through inference networks, amortizing the cost of inference and relieving the need for second derivatives during training. 3) We evaluate VERSA on benchmark datasets where the method sets new state-of-the-art results, handles arbitrary numbers of shots, and for classification, arbitrary numbers of classes at train and test time. The power of the approach is then demonstrated through a challenging few-shot ShapeNet view reconstruction task.

Meta-learning with differentiable closed-form solvers

Abstract: Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures. Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent. Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently. In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning. The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data. This requires back-propagating errors through the solver steps. While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage. We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components. Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

Meta-Learning With Differentiable Convex Optimization

Abstract: Many meta-learning approaches for few-shot learning rely on simple base learners such as nearest-neighbor classifiers. However, even in the few-shot regime, discriminatively trained linear predictors can offer better generalization. We propose to use these predictors as base learners to learn representations for few-shot learning and show they offer better tradeoffs between feature size and performance across a range of few-shot recognition benchmarks. Our objective is to learn feature embeddings that generalize well under a linear classification rule for novel categories. To efficiently solve the objective, we exploit two properties of linear classifiers: implicit differentiation of the optimality conditions of the convex problem and the dual formulation of the optimization problem. This allows us to use high-dimensional embeddings with improved generalization at a modest increase in computational overhead. Our approach, named MetaOptNet, achieves state-of-the-art performance on miniImageNet, tieredImageNet, CIFAR-FS, and FC100 few-shot learning benchmarks.

LGM-Net: Learning to Generate Matching Networks for Few-Shot Learning

Abstract: In this work, we propose a novel meta-learning approach for few-shot classification, which learns transferable prior knowledge across tasks and directly produces network parameters for similar unseen tasks with training samples. Our approach, called LGM-Net, includes two key modules, namely, TargetNet and MetaNet. The TargetNet module is a neural network for solving a specific task and the MetaNet module aims at learning to generate functional weights for TargetNet by observing training samples. We also present an intertask normalization strategy for the training process to leverage common information shared across different tasks. The experimental results on Omniglot and miniImageNet datasets demonstrate that LGM-Net can effectively adapt to similar unseen tasks and achieve competitive performance, and the results on synthetic datasets show that transferable prior knowledge is learned by the MetaNet module via mapping training data to functional weights. LGM-Net enables fast learning and adaptation since no further tuning steps are required compared to other meta-learning approaches.

Summary of Learn-to-Parameterize FSL approaches.

Many L2P approaches have been proposed in recent years, which developed various parameter generation modules to parameterize different parts of base learner. They might parameterize the task-specific predictor in base learner {{cite:0a23e60}}, {{cite:4716190}}, {{cite:df33aa5}}, {{cite:88c94ec}}, {{cite:e37378e}}, {{cite:ae7c221}}, {{cite:030a9ce}}, or the intermediate feature extraction layers in base learner {{cite:04e0056}}, {{cite:574fd79}}, {{cite:96e518c}}, even or the whole base learner {{cite:e2b3982}}, {{cite:263a4e3}}. The characteristics of different L2P approaches are summarized in Table {{table:d35bd074-d4c9-4c76-8dd4-5f8b4979a163}} .


Matching Networks for One Shot Learning

Abstract: Learning from a few examples remains a key challenge in machine learning. Despite recent advances in important domains such as vision and language, the standard supervised deep learning paradigm does not offer a satisfactory solution for learning new concepts rapidly from little data. In this work, we employ ideas from metric learning based on deep neural features and from recent advances that augment neural networks with external memories. Our framework learns a network that maps a small labelled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types. We then define one-shot learning problems on vision (using Omniglot, ImageNet) and language tasks. Our algorithm improves one-shot accuracy on ImageNet from 87.6% to 93.2% and from 88.0% to 93.8% on Omniglot compared to competing approaches. We also demonstrate the usefulness of the same model on language modeling by introducing a one-shot task on the Penn Treebank.

One-Shot Learning for Semantic Segmentation

Abstract: Low-shot learning methods for image classification support learning from sparse data. We extend these techniques to support dense semantic image segmentation. Specifically, we train a network that, given a small set of annotated images, produces parameters for a Fully Convolutional Network (FCN). We use this FCN to perform dense pixel-level prediction on a test image for the new semantic class. Our architecture shows a 25% relative meanIoU improvement compared to the best baseline methods for one-shot segmentation on unseen classes in the PASCAL VOC 2012 dataset and is at least 3 times faster.

Paris-Lille-3D: A large and high-quality ground-truth urban point cloud dataset for automatic segmentation and classification

Abstract: This paper introduces a new urban point cloud dataset for automatic segmentation and classification acquired by mobile laser scanning (MLS). We describe how the dataset is obtained from acquisition to post-processing and labeling. This dataset can be used to train pointwise classification algorithms; however, given that a great attention has been paid to the split between the different objects, this dataset can also be used to train the detection and segmentation of objects. The dataset consists of around 2 k m of MLS point cloud acquired in two cities. The number of points and range of classes mean that it can be used to train deep-learning methods. In addition, we show some results of automatic segmentation and classification. The dataset is available at: http://caor-mines-paristech.fr/fr/paris-lille-3d-dataset/.

Modeling Uncertainty with Hedged Instance Embedding

Abstract: Instance embeddings are an efficient and versatile image representation that facilitates applications like recognition, verification, retrieval, and clustering. Many metric learning methods represent the input as a single point in the embedding space. Often the distance between points is used as a proxy for match confidence. However, this can fail to represent uncertainty arising when the input is ambiguous, e.g., due to occlusion or blurriness. This work addresses this issue and explicitly models the uncertainty by hedging the location of each input in the embedding space. We introduce the hedged instance embedding (HIB) in which embeddings are modeled as random variables and the model is trained under the variational information bottleneck principle. Empirical results on our new N-digit MNIST dataset show that our method leads to the desired behavior of hedging its bets across the embedding space upon encountering ambiguous inputs. This results in improved performance for image matching and classification tasks, more structure in the learned embedding space, and an ability to compute a per-exemplar uncertainty measure that is correlated with downstream performance.

The latest performance of FSL in the main tasks of machine vision

As shown in Fig. {{figure:8c37902b-96f7-451a-9c2f-8c4aee24c563}} , during 2017-2021, 898 papers used the CUB-200-2011 {{cite:ace4bfd}} dataset, accounting for 46.6% of the total number of statistics; 587 papers used the Mini-ImageNet {{cite:edca62a}} dataset, accounting for 30.5%; and 335 papers using the Omniglot {{cite:9463f16}} dataset, accounting for 17.4% ; 44 papers used the PASCAL-5i {{cite:79e59dc}} dataset, and 46 papers used the Meta-DataSet {{cite:fd183b4}}. The other specific datasets are Paris-Lille-3D {{cite:4c6f92e}}, N-Digit MNIST {{cite:ed95e67}}, SUN397 {{cite:fb890e0}}, which are used in 15 papers in the past five years. In terms of quantity, the CUB-200-2011, Mini-ImageNet, and Omniglot benchmark datasets occupy a dominant position in the field of FSL. Table. {{table:7a93f7e9-be28-44aa-9d21-281d067ccea4}}  compares the datasets mentioned above from different dimensions. By the publication of the article, a more objective benchmark dataset {{cite:3e4807b}} for evaluating the cross-domain ability of FSL was proposed. 1) CropDiseases {{cite:8efb837}}, a plant diseases
dataset , 2) EuroSAT {{cite:1d8cea6}}, a dataset for satellite images, 3) ISIC {{cite:eb6618a}}
a medical skin image dataset, 4) ChestX {{cite:5bd742a}}, a dataset for X-ray
chest images. The similarity comparing to MiniImageNet is decreas across these datasets.


Optimal redundancy in computations from random oracles

Abstract: A classic result in algorithmic information theory is that every infinite binary sequence is computable from a Martin-Loef random infinite binary sequence. Proved independently by Kucera and Gacs, this result answered a question by Charles Bennett and has seen numerous applications in the last 30 years. The optimal redundancy in such a coding process has, however, remained unknown. If the computation of the first n bits of a sequence requires n + g(n) bits of the random oracle, then g is the redundancy of the computation. Kucera implicitly achieved redundancy n log n while Gacs used a more elaborate block-coding procedure which achieved redundancy sqrt(n) log n. Different approaches to coding such as the one by Merkle and Mihailovic have not improved this redundancy bound. In this paper we devise a new coding method that achieves optimal logarithmic redundancy. Our redundancy bound is exponentially smaller than the previously best known bound and is known to be the best possible. It follows that redundancy r log n in computation from a random oracle is possible for every stream, if and only if r > 1.

Compression of data streams down to their information content

Abstract: According to Kolmogorov complexity, every finite binary string is compressible to a shortest code -- its information content -- from which it is effectively recoverable. We investigate the extent to which this holds for infinite binary sequences (streams). We devise a new coding method which uniformly codes every stream $X$ into an algorithmically random stream $Y$, in such a way that the first $n$ bits of $X$ are recoverable from the first $I(X\upharpoonright_n)$ bits of $Y$, where $I$ is any partial computable information content measure which is defined on all prefixes of $X$, and where $X\upharpoonright_n$ is the initial segment of $X$ of length $n$. As a consequence, if $g$ is any computable upper bound on the initial segment prefix-free complexity of $X$, then $X$ is computable from an algorithmically random $Y$ with oracle-use at most $g$. Alternatively (making no use of such a computable bound $g$) one can achieve an oracle-use bounded above by $K(X\upharpoonright_n)+\log n$. This provides a strong analogue of Shannon's source coding theorem for algorithmic information theory.

Oracle use for coding by random sequences. The words “almost optimal” mean that there are two results that show that the bound is almost tight, with {{formula:32c09971-bba2-40d2-895a-b6c7bd6b148c}}  overhead or with computable upper bounds, but the conjecture mentioned above is not proven.

We gather the results mentioned above in Tables {{table:067c0079-5b6f-490c-a863-bf57f587a91a}}  and {{table:22233ecd-efc1-41a5-85ea-a89654108d96}} . We see that (for the case of oblivious coding) the optimal oblivious overhead can be obtained by replacing a local injective coding used in {{cite:a33f977}}, {{cite:eff4d5f}} by some non-local scheme. It seems that non-locality may be necessary even for getting {{formula:ba667427-0020-4728-9348-90aff289d31a}}  overhead.


Reveal a fundamental relationship between causal and noncausal MMSEs

Extend I-MMSE relationship in {{cite:9ce0e9c}} to the abstract Wiener space (scalar version)

Extend I-MMSE relationship in {{cite:9ce0e9c}} to the abstract Wiener space (vector version)

with respect to SNR and input distribution

evaluate
the mutual information and the MMSE

Extend the I-MMSE relationship to channels with feedback

Mutual information and minimum mean-square error in Gaussian channels

Abstract: This paper deals with arbitrarily distributed finite-power input signals observed through an additive Gaussian noise channel. It shows a new formula that connects the input-output mutual information and the minimum mean-square error (MMSE) achievable by optimal estimation of the input given the output. That is, the derivative of the mutual information (nats) with respect to the signal-to-noise ratio (SNR) is equal to half the MMSE, regardless of the input statistics. This relationship holds for both scalar and vector signals, as well as for discrete-time and continuous-time noncausal MMSE estimation. This fundamental information-theoretic result has an unexpected consequence in continuous-time nonlinear estimation: For any input signal with finite power, the causal filtering MMSE achieved at SNR is equal to the average value of the noncausal smoothing MMSE achieved with a channel whose SNR is chosen uniformly distributed between 0 and SNR.

On mutual information, likelihood-ratios and estimation error for the additive Gaussian channel

Abstract: This paper considers the model of an arbitrary distributed signal x observed through an added independent white Gaussian noise w, y=x+w. New relations between the minimal mean square error of the non-causal estimator and the likelihood ratio between y and \omega are derived. This is followed by an extended version of a recently derived relation between the mutual information I(x;y) and the minimal mean square error. These results are applied to derive infinite dimensional versions of the Fisher information and the de Bruijn identity. The derivation of the results is based on the Malliavin calculus.

Some relations between mutual information and estimation error in Wiener space

Abstract: The model considered is that of ``signal plus white noise.'' Known connections between the noncausal filtering error and mutual information are combined with new ones involving the causal estimation error, in a general abstract setup. The results are shown to be invariant under a wide class of causality patterns; they are applied to the derivation of the causal estimation error of a Gaussian nonstationary filtering problem and to a multidimensional extension of the Yovits--Jackson formula.

Estimation in Gaussian Noise: Properties of the Minimum Mean-Square Error

Abstract: Consider the minimum mean-square error (MMSE) of estimating an arbitrary random variable from its observation contaminated by Gaussian noise. The MMSE can be regarded as a function of the signal-to-noise ratio (SNR) as well as a functional of the input distribution (of the random variable to be estimated). It is shown that the MMSE is concave in the input distribution at any given SNR. For a given input distribution, the MMSE is found to be infinitely differentiable at all positive SNR, and in fact a real analytic function in SNR under mild conditions. The key to these regularity results is that the posterior distribution conditioned on the observation through Gaussian channels always decays at least as quickly as some Gaussian density. Furthermore, simple expressions for the first three derivatives of the MMSE with respect to the SNR are obtained. It is also shown that, as functions of the SNR, the curves for the MMSE of a Gaussian input and that of a non-Gaussian input cross at most once over all SNRs. These properties lead to simple proofs of the facts that Gaussian inputs achieve both the secrecy capacity of scalar Gaussian wiretap channels and the capacity of scalar Gaussian broadcast channels, as well as a simple proof of the entropy power inequality in the special case where one of the variables is Gaussian.

On MMSE Properties and I-MMSE Implications in Parallel MIMO Gaussian Channels

Abstract: This paper extends the "single crossing point" property of the scalar MMSE function, derived by Guo, Shamai and Verd\'u (first presented in ISIT 2008), to the parallel degraded MIMO scenario. It is shown that the matrix Q(t), which is the difference between the MMSE assuming a Gaussian input and the MMSE assuming an arbitrary input, has, at most, a single crossing point for each of its eigenvalues. Together with the I-MMSE relationship, a fundamental connection between Information Theory and Estimation Theory, this new property is employed to derive results in Information Theory. As a simple application of this property we provide an alternative converse proof for the broadcast channel (BC) capacity region under covariance constraint in this specific setting.

Statistical Physics of Signal Estimation in Gaussian Noise: Theory and Examples of Phase Transitions

Abstract: We consider the problem of signal estimation (denoising) from a statistical mechanical perspective, using a relationship between the minimum mean square error (MMSE), of estimating a signal, and the mutual information between this signal and its noisy version. The paper consists of essentially two parts. In the first, we derive several statistical-mechanical relationships between a few important quantities in this problem area, such as the MMSE, the differential entropy, the Fisher information, the free energy, and a generalized notion of temperature. We also draw analogies and differences between certain relations pertaining to the estimation problem and the parallel relations in thermodynamics and statistical physics. In the second part of the paper, we provide several application examples, where we demonstrate how certain analysis tools that are customary in statistical physics, prove useful in the analysis of the MMSE. In most of these examples, the corresponding statistical-mechanical systems turn out to consist of strong interactions that cause phase transitions, which in turn are reflected as irregularities and discontinuities (similar to threshold effects) in the behavior of the MMSE.

Pointwise relations between information and estimation in Gaussian noise

Abstract: Many of the classical and recent relations between information and estimation in the presence of Gaussian noise can be viewed as identities between expectations of random quantities. These include the I-MMSE relationship of Guo et al.; the relative entropy and mismatched estimation relationship of Verdu; the relationship between causal estimation and mutual information of Duncan, and its extension to the presence of feedback by Kadota et al.; the relationship between causal and non-casual estimation of Guo et al., and its mismatched version of Weissman. We dispense with the expectations and explore the nature of the pointwise relations between the respective random quantities. The pointwise relations that we find are as succinctly stated as - and give considerable insight into - the original expectation identities. As an illustration of our results, consider Duncan's 1970 discovery that the mutual information is equal to the causal MMSE in the AWGN channel, which can equivalently be expressed saying that the difference between the input-output information density and half the causal estimation error is a zero mean random variable (regardless of the distribution of the channel input). We characterize this random variable explicitly, rather than merely its expectation. Classical estimation and information theoretic quantities emerge with new and surprising roles. For example, the variance of this random variable turns out to be given by the causal MMSE (which, in turn, is equal to the mutual information by Duncan's result).

Extensions of the I-MMSE relation

Abstract: Unveiling a fundamental link between information theory and estimation theory, the I-MMSE relation by Guo, Shamai and Verdú [4] has great theoretical significance and numerous practical applications. On the other hand, its influences to date have been restricted to channels without feedback or memory, due to the lack of extensions of the I-MMSE relation to such channels. In this paper, we propose extensions of the I-MMSE relation for discrete and continuous-time Gaussian channels with feedback or memory. Our approach is based on a very simple observation, which can be applied to other scenarios, such as a simple and direct proof of the classical de Bruijn's identity.

Extensions of the I-MMSE relation

Recent research results of the I-MMSE relationships for Gaussian channel

where {{formula:cac1f7d3-416d-4e61-a931-79f9e8f8f2c7}}  is the relative entropy.
{{formula:17793353-cc31-4900-9d0f-de85fa1e5d8d}}  and {{formula:08a88a92-694b-44b7-8a09-9cbbcbfb8dc9}}
are MMSEs obtained by an estimator which assumes the input signal is distributed
according to {{formula:1a919c56-ce40-4de9-b7c1-3d04132e20af}}  and {{formula:d78a82b5-d451-436e-b0a1-c665e29b0cc7}}  for a real-value scalar Gaussian channel, respectively.
Eq. (REF ) is a generalization of Eq. (REF ) for the scalar Gaussian channel
with a mismatch estimation. Similarly, using the mutual information and the relative entropy as a bridge,
T. Weissman generalizes Eq. (REF ) into the case of mismatch estimation in {{cite:73dfeeb}}.
The statistic properties of the difference
between the mismatched and matched estimation losses
are studied in {{cite:e410818}}. The relationship between relative entropy and mismatched estimation
is extended to the real-value vector Gaussian channel in {{cite:e51f28c}}. For a more complicated
case of mismatched estimation where the channel distribution is mismatched at the estimator instead
of the input signal distribution, W. Huleihel et al. derive the mismatched MSE for the
vector Gaussian channel using the statistical physics tools {{cite:5bf1f9a}}.
A brief summation
of recent research results
of the I-MMSE relationships for Gaussian channel
is given in Table {{table:77b6be17-5681-4f65-a590-90f28ca11752}} .


EcoCyc: fusing model organism databases with systems biology

Abstract: EcoCyc (http://EcoCyc.org) is a model organism database built on the genome sequence of Escherichia coli K-12 MG1655. Expert manual curation of the functions of individual E. coli gene products in EcoCyc has been based on information found in the experimental literature for E. coli K-12-derived strains. Updates to EcoCyc content continue to improve the comprehensive picture of E. coli biology. The utility of EcoCyc is enhanced by new tools available on the EcoCyc web site, and the development of EcoCyc as a teaching tool is increasing the impact of the knowledge collected in EcoCyc.

TBestDB: a taxonomically broad database of expressed sequence tags (ESTs)

Abstract: The TBestDB database contains approximately 370,000 clustered expressed sequence tag (EST) sequences from 49 organisms, covering a taxonomically broad range of poorly studied, mainly unicellular eukaryotes, and includes experimental information, consensus sequences, gene annotations and metabolic pathway predictions. Most of these ESTs have been generated by the Protist EST Program, a collaboration among six Canadian research groups. EST sequences are read from trace files up to a minimum quality cut-off, vector and linker sequence is masked, and the ESTs are clustered using phrap. The resulting consensus sequences are automatically annotated by using the AutoFACT program. The datasets are automatically checked for clustering errors due to chimerism and potential cross-contamination between organisms, and suspect data are flagged in or removed from the database. Access to data deposited in TBestDB by individual users can be restricted to those users for a limited period. With this first report on TBestDB, we open the database to the research community for free processing, annotation, interspecies comparisons and GenBank submission of EST data generated in individual laboratories. For instructions on submission to TBestDB, contact tbestdb@bch.umontreal.ca. The database can be queried at http://tbestdb.bcm.umontreal.ca/.

PGDBs for microbes.

Survey of Pathway Tools Compatible Databases

Available PGDBs include those in Tables
{{table:3c659382-c050-4da4-8c89-95c9cd7f55c3}} –{{table:84b3c5c6-0157-4b0b-9f4c-c4694acac2b8}} , with curated PGDBs in
bold.


Momentum (Rumelhart et al. {{cite:3fc5f3d}})

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Abstract: Deep learning thrives with large neural networks and large datasets. However, larger networks and larger datasets result in longer training times that impede research and development progress. Distributed synchronous SGD offers a potential solution to this problem by dividing SGD minibatches over a pool of parallel workers. Yet to make this scheme efficient, the per-worker workload must be large, which implies nontrivial growth in the SGD minibatch size. In this paper, we empirically show that on the ImageNet dataset large minibatches cause optimization difficulties, but when these are addressed the trained networks exhibit good generalization. Specifically, we show no loss of accuracy when training with large minibatch sizes up to 8192 images. To achieve this result, we adopt a hyper-parameter-free linear scaling rule for adjusting learning rates as a function of minibatch size and develop a new warmup scheme that overcomes optimization challenges early in training. With these simple techniques, our Caffe2-based system trains ResNet-50 with a minibatch size of 8192 on 256 GPUs in one hour, while matching small minibatch accuracy. Using commodity hardware, our implementation achieves ~90% scaling efficiency when moving from 8 to 256 GPUs. Our findings enable training visual recognition models on internet-scale data with high efficiency.

Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

Abstract: We demonstrate that training ResNet-50 on ImageNet for 90 epochs can be achieved in 15 minutes with 1024 Tesla P100 GPUs. This was made possible by using a large minibatch size of 32k. To maintain accuracy with this large minibatch size, we employed several techniques such as RMSprop warm-up, batch normalization without moving averages, and a slow-start learning rate schedule. This paper also describes the details of the hardware and software of the system used to achieve the above performance.

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

Abstract: Synchronized stochastic gradient descent (SGD) optimizers with data parallelism are widely used in training large-scale deep neural networks. Although using larger mini-batch sizes can improve the system scalability by reducing the communication-to-computation ratio, it may hurt the generalization ability of the models. To this end, we build a highly scalable deep learning training system for dense GPU clusters with three main contributions: (1) We propose a mixed-precision training method that significantly improves the training throughput of a single GPU without losing accuracy. (2) We propose an optimization approach for extremely large mini-batch size (up to 64k) that can train CNN models on the ImageNet dataset without losing accuracy. (3) We propose highly optimized all-reduce algorithms that achieve up to 3x and 11x speedup on AlexNet and ResNet-50 respectively than NCCL-based training on a cluster with 1024 Tesla P40 GPUs. On training ResNet-50 with 90 epochs, the state-of-the-art GPU-based system with 1024 Tesla P100 GPUs spent 15 minutes and achieved 74.9\% top-1 test accuracy, and another KNL-based system with 2048 Intel KNLs spent 20 minutes and achieved 75.4\% accuracy. Our training system can achieve 75.8\% top-1 test accuracy in only 6.6 minutes using 2048 Tesla P40 GPUs. When training AlexNet with 95 epochs, our system can achieve 58.7\% top-1 test accuracy within 4 minutes, which also outperforms all other existing systems.

ImageNet/ResNet-50 Training in 224 Seconds

Abstract: Scaling the distributed deep learning to a massive GPU cluster level is challenging due to the instability of the large mini-batch training and the overhead of the gradient synchronization. We address the instability of the large mini-batch training with batch size control. We address the overhead of the gradient synchronization with 2D-Torus all-reduce. Specifically, 2D-Torus all-reduce arranges GPUs in a logical 2D grid and performs a series of collective operation in different orientations. These two techniques are implemented with Neural Network Libraries (NNL). We have successfully trained ImageNet/ResNet-50 in 224 seconds without significant accuracy loss on ABCI cluster.

Image Classification at Supercomputer Scale

Abstract: Deep learning is extremely computationally intensive, and hardware vendors have responded by building faster accelerators in large clusters. Training deep learning models at petaFLOPS scale requires overcoming both algorithmic and systems software challenges. In this paper, we discuss three systems-related optimizations: (1) distributed batch normalization to control per-replica batch sizes, (2) input pipeline optimizations to sustain model throughput, and (3) 2-D torus all-reduce to speed up gradient summation. We combine these optimizations to train ResNet-50 on ImageNet to 76.3% accuracy in 2.2 minutes on a 1024-chip TPU v3 Pod with a training throughput of over 1.05 million images/second and no accuracy drop.

Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 seconds

Abstract: There has been a strong demand for algorithms that can execute machine learning as faster as possible and the speed of deep learning has accelerated by 30 times only in the past two years. Distributed deep learning using the large mini-batch is a key technology to address the demand and is a great challenge as it is difficult to achieve high scalability on large clusters without compromising accuracy. In this paper, we introduce optimization methods which we applied to this challenge. We achieved the training time of 74.7 seconds using 2,048 GPUs on ABCI cluster applying these methods. The training throughput is over 1.73 million images/sec and the top-1 validation accuracy is 75.08%.

Scale MLPerf-0.6 models on Google TPU-v3 Pods

Abstract: The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defines a suite of models, datasets and rules to follow when benchmarking to ensure results are comparable across hardware, frameworks and companies. Using this suite of models, we discuss the optimizations and techniques including choice of optimizer, spatial partitioning and weight update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify properties of models that make scaling them challenging, such as limited data parallelism and unscaled weights. These optimizations contribute to record performance in transformer, Resnet-50 and SSD in the Google MLPerf-0.6 submission.

ImageNet/ResNet-50 Training Speed Records.

Large-batch optimization is becoming an important research topic as it is an efficient approach for current distributed deep learning systems.
It has enabled researchers to reduce the ImageNet/ResNet-50 training from 29 hours to around 1 minute (Table {{table:89cfa3e7-b941-4885-9c29-1e741286f1b4}} ).
Researchers can also reduce the BERT pre-training time from 3 days to 76 minutes {{cite:ce48eaf}}, 67 minutes {{cite:eed4554}}, 47 minutes {{cite:7f8da2f}} and 44 minutes {{cite:c2f4df5}}.
The speedup comes from the fact that researchers can scale the training to a larger batch size, and so use larger scale, parallelism, without losing accuracy in a fixed number of epochs.
For example, researchers scaled the batch size of ImageNet training from 256 {{cite:9ea10e8}} to 1K {{cite:6077f12}}, 5K {{cite:72d41f8}}, 8K {{cite:79995f7}}, 16K {{cite:b9f2f56}}, 32K {{cite:2c5f1d5}}, and 64K {{cite:d0f76eb}}.
On the other hand, the hardware designers are also interested in large-batch training, as the chip makers fail to continue improving the single processor's clock frequency due to power consumption issue {{cite:d4dbee4}}.
Thus, the hardware vendors have to use more processors to increase the raw floating point computing performance, which requires high-parallelism for future algorithms.
Many successful use cases of large-batch training have encouraged industry to design and implement several AI supercomputers like Google TPU Pod, NVIDIA SuperPOD, and Huawei Atlas 900 AI Cluster to take advantage of the extremely high parallelism provided by large-batch algorithms.

Keskar et al. {{cite:60b7494}} reported that traditional first-order optimization techniques fail to scale up the batch size to a very large number.
Large-batch training will hurt the testing accuracy.
They indicate there is an inherent generalization gap in large-batch optimization within fixed number of epochs.
Hoffer et al. {{cite:f651451}} suggested training longer can close the generalization gap.
After that, researchers have developed a series of optimization techniques and scaled the batch size to 64K without losing accuracy for various applications within fixed number of epochs (Table {{table:89cfa3e7-b941-4885-9c29-1e741286f1b4}} ).
There are also other related literatures {{cite:8aafe83}} {{cite:c0cad69}} {{cite:cf523c6}} {{cite:3e602dd}} {{cite:6077f12}} {{cite:457f6ac}} {{cite:a1d50ba}} {{cite:f81c3d1}} {{cite:6af0f62}} {{cite:a0e6145}} {{cite:85d149a}}.
Inspired by these works, we focus on a different direction.
As far as we know, there is no previous work studying huge-batch optimization (e.g. over 128K).
Our investigation includes: (1) pushing the batch size to the limit and maximizing the accuracy in fixed number of epochs; (2) studying the optimization problems of huge-batch and full-batch; (3) studying the relationship between generalization and optimization.


Natural language understanding in virtual environments, sentiment analysis, emotion detection in virtual interactions, text classification, and recommendation systems in the metaverse.

Language understanding and generation in VR scenarios, text summarization and content extraction for immersive experiences, sentiment analysis, and opinion mining in virtual social interactions.

Training large-scale language models can be computationally expensive, fine-tuning for specific metaverse domains and contexts. Handling out-of-vocabulary words and new concepts in virtual environments.

Large storage and computational requirements during training, adapting the model to different metaverse platforms and languages, mitigating potential biases in training data and generated outputs.

BFCN: A Novel Classification Method of Encrypted Traffic Based on BERT and CNN

Abstract: With the speedy advancement of encryption technology and the exponential increase in applications, network traffic classification has become an increasingly important research topic. Existing methods for classifying encrypted traffic have certain limitations. For example, traditional approaches such as machine learning rely heavily on feature engineering, deep learning approaches are susceptible to the amount and distribution of labeled data, and pretrained models focus merely on the global traffic features while ignoring local features. To solve the above problem, we propose a BERT-based byte-level feature convolutional network (BFCN) model consisting of two novel modules. The first is a packet encoder module, in which we use the BERT pretrained encrypted traffic classification model to capture global traffic features through its attention mechanism; the second is a CNN module, which captures byte-level local features in the traffic through convolutional operations. The packet-level and byte-level features are concatenated as the traffic’s final representation, which can better represent encrypted traffic. Our approach achieves state-of-the-art performance on the publicly available ISCX-VPN dataset for the traffic service and application identification task, achieving F1 scores of 99.11% and 99.41%, respectively, on these two tasks. The experimental results demonstrate that our method further improves the performance of encrypted traffic classification.

RobBERT: a Dutch RoBERTa-based Language Model

Abstract: Pre-trained language models have been dominating the field of natural language processing in recent years, and have led to significant performance gains for various complex natural language tasks. One of the most prominent pre-trained language models is BERT, which was released as an English as well as a multilingual version. Although multilingual BERT performs well on many tasks, recent studies show that BERT models trained on a single language significantly outperform the multilingual version. Training a Dutch BERT model thus has a lot of potential for a wide range of Dutch NLP tasks. While previous approaches have used earlier implementations of BERT to train a Dutch version of BERT, we used RoBERTa, a robustly optimized BERT approach, to train a Dutch language model called RobBERT. We measured its performance on various tasks as well as the importance of the fine-tuning dataset size. We also evaluated the importance of language-specific tokenizers and the model's fairness. We found that RobBERT improves state-of-the-art results for various tasks, and especially significantly outperforms other models when dealing with smaller datasets. These results indicate that it is a powerful pre-trained model for a large variety of Dutch language tasks. The pre-trained and fine-tuned models are publicly available to support further downstream Dutch NLP applications.

Comparison and decomposition of foundation models, their potential applications in the metaverse, and challenges.

AI applications in the metaverse, such as machine vision, generative modeling, speech processing, and NLP {{cite:2625f69}}, {{cite:c65accb}}, offer enhanced user experiences and foster new creativity and innovation. Generative modeling allows the creation of unique virtual environments, characters, and objects, facilitating efficient content generation based on existing data. Speech processing enables voice chat for more natural and immersive interactions between users, facilitating collaboration and socialization in virtual environments. NLP powers chatbots and virtual assistants that help users navigate the metaverse, perform tasks, and provide information, making it user-friendly, especially for newcomers. For a comparison and decomposition of foundation models, their potential metaverse applications, and challenges, refer to Table {{table:a0f83ab7-461a-4766-8ff4-02278d66499c}} .


Very Deep Convolutional Networks for Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Rethinking the Inception Architecture for Computer Vision

Abstract: Convolutional networks are at the core of most state-of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we explore ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21.2% top-1 and 5.6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3.5% top-5 error on the validation set (3.6% error on the test set) and 17.3% top-1 error on the validation set.

Directional Message Passing for Molecular Graphs

Abstract: Graph neural networks have recently achieved great successes in predicting quantum mechanical properties of molecules. These models represent a molecule as a graph using only the distance between atoms (nodes). They do not, however, consider the spatial direction from one atom to another, despite directional information playing a central role in empirical potentials for molecules, e.g. in angular potentials. To alleviate this limitation we propose directional message passing, in which we embed the messages passed between atoms instead of the atoms themselves. Each message is associated with a direction in coordinate space. These directional message embeddings are rotationally equivariant since the associated directions rotate with the molecule. We propose a message passing scheme analogous to belief propagation, which uses the directional information by transforming messages based on the angle between them. Additionally, we use spherical Bessel functions and spherical harmonics to construct theoretically well-founded, orthogonal representations that achieve better performance than the currently prevalent Gaussian radial basis representations while using fewer than 1/4 of the parameters. We leverage these innovations to construct the directional message passing neural network (DimeNet). DimeNet outperforms previous GNNs on average by 76% on MD17 and by 31% on QM9. Our implementation is available online.

SchNet: A continuous-filter convolutional neural network for modeling quantum interactions

Abstract: Deep learning has the potential to revolutionize quantum chemistry as it is ideally suited to learn representations for structured data and speed up the exploration of chemical space. While convolutional neural networks have proven to be the first choice for images, audio and video data, the atoms in molecules are not restricted to a grid. Instead, their precise locations contain essential physical information, that would get lost if discretized. Thus, we propose to use continuous-filter convolutional layers to be able to model local correlations without requiring the data to lie on a grid. We apply those layers in SchNet: a novel deep learning architecture modeling quantum interactions in molecules. We obtain a joint model for the total energy and interatomic forces that follows fundamental quantum-chemical principles. This includes rotationally invariant energy predictions and a smooth, differentiable potential energy surface. Our architecture achieves state-of-the-art performance for benchmarks of equilibrium molecules and molecular dynamics trajectories. Finally, we introduce a more challenging benchmark with chemical and structural variations that suggests the path for further work.

Deep Neural Network Models

In this paper, we train six different, representative DL models (Table {{table:50f8a47f-1308-4d59-99e8-294959011183}} ) with applications across computer vision (CV), natural language processing (NLP), and geometric deep learning (GDL) and investigate their scaling behavior across hundreds of GPUs. We monitor GPU utilization and energy consumption during distributed training and identify optimal settings for efficient training and opportunities for improved scaling and hardware utilization. Our main goal is not to generate precise predictions of execution time, but instead to study the impacts of and the relationship between model architecture and compute utilization on distributed training time. By comparing model architectures via their scaling exponents, we can estimate training times for variations on common architectures such as convolutional neural networks (CNNs), transformer-based language models, and graph neural networks (GNNs). This will help scientific DL practitioners in developing methods to better profile different model architectures and determine the most time and energy-efficient workflow for their own hardware configurations.

Our choices of benchmarks and models were informed by our operational experiences from the workloads and usage patterns characteristic of the system—we chose DNN models and training tasks that are representative of common user workloads, but also models that tend to be frequent workhorses for research, testing, and experiments in the broader research community. We also sought to capture a diverse range of "classic" and state-of-the-art DL architectures that are widely used across different domains. Our intent in this work is not to propose a new benchmark or evaluate the relative merit of one benchmark against another. Rather, our goal is to investigate and illustrate the scaling behavior of representative model architectures, based on computational workloads on an operational supercomputing system, and develop simple descriptions of training execution time. The specific models in our experiments are listed in Table {{table:50f8a47f-1308-4d59-99e8-294959011183}}  and can be grouped into three domains or categories characterized by the commonality of their tasks, similarity of model architectures, or both.

Throughout our experiments, for each model in Table {{table:50f8a47f-1308-4d59-99e8-294959011183}} , we collect time-series data on their GPU memory utilization and streaming multiprocessor (SM) utilization, training speed, and total energy consumption/expenditure throughout each of their training runs under different GPU power caps (100 W, 200 W, and 250 W), GPU clock rates (135 MHz, 735 MHz, and 1380 MHz), and number of GPUs (2, 4, 8, 16, 32, 64, etc.) aggregated on a per-epoch level. When a power or clock rate cap is applied, it is applied uniformly and simultaneously to all GPUs used in training. Model training was performed on NVIDIA Tesla V100 GPUs and GPU utilization; power consumption etc., were monitored using the nvidia-smi tool. Training speed or training time is calculated as wall-clock time per epoch, measured in seconds, and power/energy consumption (or training energy) measured in joules, which is calculated by multiplying wall-clock time spent in training by the power consumption measured in watts.

When scaling up the number of GPUs, changes in GPU-utilization are heavily architecture-dependent, even within domains/model classes; however, all models show reduced memory utilization during training. Naturally, different models are likely to exhibit different GPU utilization profiles and these differences are likely to become more pronounced as differences in network architecture, and therefore computational pathways, become more pronounced. In Figure  {{figure:84a3573b-7318-48b5-b63d-3e7cae9e4f7b}} , we show the joint GPU memory and SM utilization characteristics for the models described in Table {{table:50f8a47f-1308-4d59-99e8-294959011183}}  at two representative scales of distributed training (on 2 GPUs versus 128 GPUs). Geometric models are shown in blue, BERT is shown in red, and vision models are shown in teal, where memory and SM utilization are averaged over all GPU workers.