Going Deeper with Convolutions

Abstract: We propose a deep convolutional neural network architecture codenamed "Inception", which was responsible for setting the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC 2014). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. This was achieved by a carefully crafted design that allows for increasing the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC 2014 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Abstract: We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.

WaveNet: A Generative Model for Raw Audio

Abstract: This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-of-the-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.

8 popular deep learning models used in this work, as well as their usage in applications and input data format.

Deep learning (DL) is now having transformative impacts on how mobile sensor data are utilized and interpreted.
In recent years,
Many new mobile applications have been developed with the capability of deep learning {{cite:a2db7ed}} {{cite:6c3e16f}} {{cite:4bc8ff8}} {{cite:c823e7a}} {{cite:1d02bff}}.
Also many DL frameworks and specialized hardware devices for speeding up DL have been proposed {{cite:6ada0a2}} {{cite:ecaae74}} {{cite:fc604de}} {{cite:ea1fb21}} {{cite:503ed2a}}.
Some of DL's representative use cases are shown in Table {{table:1a30b585-de46-43a6-ab1f-1d195c170401}} .
However, running such DL tasks on wearables is quite challenging due to wearables' unique characteristics such as weak computation power, tight energy constraints, and short-range pairing with smartphones.


Weakly-Supervised Classification of HER2 Expression in Breast Cancer Haematoxylin and Eosin Stained Slides

Abstract: Human epidermal growth factor receptor 2 (HER2) evaluation commonly requires immunohistochemistry (IHC) tests on breast cancer tissue, in addition to the standard haematoxylin and eosin (H&E) staining tests. Additional costs and time spent on further testing might be avoided if HER2 overexpression could be effectively inferred from H&E stained slides, as a preliminary indication of the IHC result. In this paper, we propose the first method that aims to achieve this goal. The proposed method is based on multiple instance learning (MIL), using a convolutional neural network (CNN) that separately processes H&E stained slide tiles and outputs an IHC label. This CNN is pretrained on IHC stained slide tiles but does not use these data during inference/testing. H&E tiles are extracted from invasive tumour areas segmented with the HASHI algorithm. The individual tile labels are then combined to obtain a single label for the whole slide. The network was trained on slides from the HER2 Scoring Contest dataset (HER2SC) and tested on two disjoint subsets of slides from the HER2SC database and the TCGA-TCIA-BRCA (BRCA) collection. The proposed method attained 83.3 % classification accuracy on the HER2SC test set and 53.8 % on the BRCA test set. Although further efforts should be devoted to achieving improved performance, the obtained results are promising, suggesting that it is possible to perform HER2 overexpression classification on H&E stained tissue slides.

A Pyramid Architecture-Based Deep Learning Framework for Breast Cancer Detection

Abstract: Breast cancer diagnosis is a critical step in clinical decision making, and this is achieved by making a pathological slide and gives a decision by the doctors, which is the method of final decision making for cancer diagnosis. Traditionally, the doctors usually check the pathological images by visual inspection under the microscope. Whole-slide images (WSIs) have supported the state-of-the-art diagnosis results and have been admitted as the gold standard clinically. However, this task is time-consuming and labour-intensive, and all of these limitations make low efficiency in decision making. Medical image processing protocols have been used for this task during the last decades and have obtained satisfactory results under some conditions; especially in the deep learning era, it has exhibited the advantages than those in the shallow learning period. In this paper, we proposed a novel breast cancer region mining framework based on deep pyramid architecture from multilevel and multiscale breast pathological WSIs. We incorporate the tissue- and cell-level information together and integrate these into a LSTM model for the final sequence modelling, which successfully keeps the WSIs' integration and is not mentioned by the prevalence frameworks. The experiment results demonstrated that our proposed framework greatly improved the detection accuracy than that only using tissue-level information.

Automated Quantification of sTIL Density with H&E-Based Digital Image Analysis Has Prognostic Potential in Triple-Negative Breast Cancers

Abstract: Simple Summary Around 15% of breast cancer patients are diagnosed as triple-negative (TNBC), which have significantly lower 5-year survival rates (77%) than other types of breast cancer (93%). Our study aimed at developing an image analysis-based biomarker to assess how the immune system interacts with the tumor and investigate the potential added value of stromal tumor-infiltrating lymphocytes (sTIL) for the prognosis of overall survival compared to the manual approach. In a large retrospective cohort of 257 patients, we found that our fully automated hematoxylin and eosin (H&E) image analysis pipeline can quantify sTIL density showing both high concordance with manual scoring and association with the prognosis of patients with TNBC. It also overcomes natural limitations of manual assessment that hinder clinical adoption of the immune biomarker. We conclude that sTIL scoring by automated image analysis has prognostic potential comparable to manual scoring and should be further investigated for future use in a clinical setting. Abstract Triple-negative breast cancer (TNBC) is an aggressive and difficult-to-treat cancer type that represents approximately 15% of all breast cancers. Recently, stromal tumor-infiltrating lymphocytes (sTIL) resurfaced as a strong prognostic biomarker for overall survival (OS) for TNBC patients. Manual assessment has innate limitations that hinder clinical adoption, and the International Immuno-Oncology Biomarker Working Group (TIL-WG) has therefore envisioned that computational assessment of sTIL could overcome these limitations and recommended that any algorithm should follow the manual guidelines where appropriate. However, no existing studies capture all the concepts of the guideline or have shown the same prognostic evidence as manual assessment. In this study, we present a fully automated digital image analysis pipeline and demonstrate that our hematoxylin and eosin (H&E)-based pipeline can provide a quantitative and interpretable score that correlates with the manual pathologist-derived sTIL status, and importantly, can stratify a retrospective cohort into two significant distinct prognostic groups. We found our score to be prognostic for OS (HR: 0.81 CI: 0.72–0.92 p = 0.001) independent of age, tumor size, nodal status, and tumor type in statistical modeling. While prior studies have followed fragments of the TIL-WG guideline, our approach is the first to follow all complex aspects, where appropriate, supporting the TIL-WG vision of computational assessment of sTIL in the future clinical setting.

Deep Learning Predicts Underlying Features on Pathology Images with Therapeutic Relevance for Breast and Gastric Cancer

Abstract: Simple Summary DNA repair deficiency (DRD) is common in many cancers. This deficiency contributes to pathogenesis of the disease, but it also presents an opportunity for therapeutic targeting. However, current DRD identification assays are not available for all patients. We propose an efficient machine learning algorithm which can predict DRD from histopathological images. The utility of our method was shown by considering the detection of homologous recombination deficiency (HRD) and mismatch repair deficiency (MMRD) in breast and gastric cancer respectively. Our findings demonstrate that machine-learning approaches can be used in advanced applications to assist therapy decisions. Abstract DNA repair deficiency (DRD) is an important driver of carcinogenesis and an efficient target for anti-tumor therapies to improve patient survival. Thus, detection of DRD in tumors is paramount. Currently, determination of DRD in tumors is dependent on wet-lab assays. Here we describe an efficient machine learning algorithm which can predict DRD from histopathological images. The utility of this algorithm is demonstrated with data obtained from 1445 cancer patients. Our method performs rather well when trained on breast cancer specimens with homologous recombination deficiency (HRD), AUC (area under curve) = 0.80. Results for an independent breast cancer cohort achieved an AUC = 0.70. The utility of our method was further shown by considering the detection of mismatch repair deficiency (MMRD) in gastric cancer, yielding an AUC = 0.81. Our results demonstrate the capacity of our learning-base system as a low-cost tool for DRD detection.

Multi-dataset utilization in the included articles

Deep learning models can benefit from external datasets to improve their ability to generalize to new data and enhance their performance on a specific task. Using external data to validate trained algorithms can help ensure their generalizability, identifying overfitting and assessing their performance across different datasets. So, the real measure of a model's predictive ability lies in its performance on an independent dataset that was not employed in its initial development {{cite:092825e}}, as the performance of these models often diminishes when applied to a new cohort beyond the original development population {{cite:070c508}}. Typically, there is a lack of external validation in the algorithms developed for breast computational pathology. Among the 139 papers analyzed in our study, only 30 papers incorporated multiple datasets, and 14 of those integrated private datasets alongside public ones for model development. Moreover, just 16 studies used an external validation set (Table {{table:cbec0df5-c40f-4cef-92bc-a57de8bf5c0e}} ), implying that the performance of other developed models could be subject to overestimation. In 11 papers, the TCGA-BRCA dataset was combined with a single private dataset to develop the models. One paper employed the combination of the Camelyon 17 with a private dataset, and one paper integrated the TCGA-BRCA, Camelyon 16 and 17, and CPTAC-BRCA datasets with a private dataset for model development.


Matterport3D: Learning from RGB-D Data in Indoor Environments

Abstract: Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Abstract: A key requirement for leveraging supervised deep learning methods is the availability of large, labeled datasets. Unfortunately, in the context of RGB-D scene understanding, very little data is available – current datasets cover a small range of scene views and have limited semantic annotations. To address this issue, we introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations. To collect this data, we designed an easy-to-use and scalable RGB-D capture system that includes automated surface reconstruction and crowdsourced semantic annotation. We show that using this data helps achieve state-of-the-art performance on several 3D scene understanding tasks, including 3D object classification, semantic voxel labeling, and CAD model retrieval.

Comparison of our proposed Cross3DVG dataset with existing 3D visual grounding datasets.

Datasets for 3D Visual Grounding.
To facilitate 3D visual grounding, we introduce the Cross3DVG dataset, which is created by combining the RIORefer dataset, annotating indoor 3D scenes from 3RScan {{cite:b104007}}, with the existing ScanRefer dataset {{cite:2ae73b9}} dataset (see details in Section ).
In Table {{table:8bc3950f-e53d-4af6-add4-0adb3a6b2daf}} , we provide a comparison between the existing 3D visual grounding datasets and our Cross3DVG dataset.
In general, 3D visual grounding datasets are mainly created by language annotating existing 3D datasets, such as Matterport3D {{cite:bcbd392}}, SUN RGB-D {{cite:fa8c819}}, and ScanNet {{cite:782a974}}.
These 3D datasets are often created using different 3D data acquisition methods (e.g., Matterport, Kinect, and Structure Sensor).
REVERIE {{cite:5e99d57}} is a large-scale remote object grounding dataset containing 21k natural language descriptions for instructing embodied agents to locate target objects in photorealistic 3D indoor environments.
SUN-Spot {{cite:7242995}} and SUNRefer {{cite:485d372}} focus on object localization within single-view RGB-D images captured inside buildings or houses.
Compared to these datasets using RGB and RGB-D images from 3D environments, Nr3D {{cite:431864b}} and ScanRefer {{cite:2ae73b9}} use dense 3D scans of ScanNet captured by Structure Sensor on an iPad.
In particular, ScanRefer contains the longest and largest number of descriptions.
However, despite real-world applications often employing various sensors, existing 3D visual grounding datasets mainly consist of 3D data acquired by a single type of 3D scanning system.
In contrast, Cross3DVG consists of two different types of 3D datasets to evaluate the generalization capabilities of 3D visual grounding methods across diverse 3D datasets.


SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models

Abstract: Single image super-resolution (SISR) aims to reconstruct high-resolution (HR) images from the given low-resolution (LR) ones, which is an ill-posed problem because one LR image corresponds to multiple HR images. Recently, learning-based SISR methods have greatly outperformed traditional ones, while suffering from over-smoothing, mode collapse or large model footprint issues for PSNR-oriented, GAN-driven and flow-based methods respectively. To solve these problems, we propose a novel single image super-resolution diffusion probabilistic model (SRDiff), which is the first diffusion-based model for SISR. SRDiff is optimized with a variant of the variational bound on the data likelihood and can provide diverse and realistic SR predictions by gradually transforming the Gaussian noise into a super-resolution (SR) image conditioned on an LR input through a Markov chain. In addition, we introduce residual prediction to the whole framework to speed up convergence. Our extensive experiments on facial and general benchmarks (CelebA and DIV2K datasets) show that 1) SRDiff can generate diverse SR results in rich details with state-of-the-art performance, given only one LR input; 2) SRDiff is easy to train with a small footprint; and 3) SRDiff can perform flexible image manipulation including latent space interpolation and content fusion.

Cascaded Diffusion Models for High Fidelity Image Generation

Abstract: We show that cascaded diffusion models are capable of generating high fidelity images on the class-conditional ImageNet generation benchmark, without any assistance from auxiliary image classifiers to boost sample quality. A cascaded diffusion model comprises a pipeline of multiple diffusion models that generate images of increasing resolution, beginning with a standard diffusion model at the lowest resolution, followed by one or more super-resolution diffusion models that successively upsample the image and add higher resolution details. We find that the sample quality of a cascading pipeline relies crucially on conditioning augmentation, our proposed method of data augmentation of the lower resolution conditioning inputs to the super-resolution models. Our experiments show that conditioning augmentation prevents compounding error during sampling in a cascaded model, helping us to train cascading pipelines achieving FID scores of 1.48 at 64x64, 3.52 at 128x128 and 4.88 at 256x256 resolutions, outperforming BigGAN-deep, and classification accuracy scores of 63.02% (top-1) and 84.06% (top-5) at 256x256, outperforming VQ-VAE-2.

Label-Efficient Semantic Segmentation with Diffusion Models

Abstract: Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance. The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem. Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision.

3D Shape Generation and Completion through Point-Voxel Diffusion

Abstract: We propose a novel approach for probabilistic generative modeling of 3D shapes. Unlike most existing models that learn to deterministically translate a latent vector to a shape, our model, Point-Voxel Diffusion (PVD), is a unified, probabilistic formulation for unconditional shape generation and conditional, multi-modal shape completion. PVD marries denoising diffusion models with the hybrid, point-voxel representation of 3D shapes. It can be viewed as a series of denoising steps, reversing the diffusion process from observed point cloud data to Gaussian noise, and is trained by optimizing a variational lower bound to the (conditional) likelihood function. Experiments demonstrate that PVD is capable of synthesizing high-fidelity shapes, completing partial point clouds, and generating multiple completion results from single-view depth scans of real objects.

Flexible Diffusion Modeling of Long Videos

Abstract: We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames. We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA autonomous driving simulator.

Structured Denoising Diffusion Models in Discrete State-Spaces

Abstract: Denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) have shown impressive results on image and waveform generation in continuous state spaces. Here, we introduce Discrete Denoising Diffusion Probabilistic Models (D3PMs), diffusion-like generative models for discrete data that generalize the multinomial diffusion model of Hoogeboom et al. 2021, by going beyond corruption processes with uniform transition probabilities. This includes corruption with transition matrices that mimic Gaussian kernels in continuous space, matrices based on nearest neighbors in embedding space, and matrices that introduce absorbing states. The third allows us to draw a connection between diffusion models and autoregressive and mask-based generative models. We show that the choice of transition matrix is an important design decision that leads to improved results in image and text domains. We also introduce a new loss function that combines the variational lower bound with an auxiliary cross entropy loss. For text, this model class achieves strong results on character-level text generation while scaling to large vocabularies on LM1B. On the image dataset CIFAR-10, our models approach the sample quality and exceed the log-likelihood of the continuous-space DDPM model.

WaveGrad: Estimating Gradients for Waveform Generation

Abstract: This paper introduces WaveGrad, a conditional model for waveform generation which estimates gradients of the data density. The model is built on prior work on score matching and diffusion probabilistic models. It starts from a Gaussian white noise signal and iteratively refines the signal via a gradient-based sampler conditioned on the mel-spectrogram. WaveGrad offers a natural way to trade inference speed for sample quality by adjusting the number of refinement steps, and bridges the gap between non-autoregressive and autoregressive models in terms of audio quality. We find that it can generate high fidelity audio samples using as few as six iterations. Experiments reveal WaveGrad to generate high fidelity audio, outperforming adversarial non-autoregressive baselines and matching a strong likelihood-based autoregressive baseline using fewer sequential operations. Audio samples are available at https://wavegrad.github.io/.

Blended Diffusion for Text-driven Editing of Natural Images

Abstract: Natural language offers a highly intuitive interface for image editing. In this paper, we introduce the first solution for performing local (region-based) edits in generic natural images, based on a natural language description along with an ROI mask. We achieve our goal by leveraging and combining a pretrained language-image model (CLIP), to steer the edit towards a user-provided text prompt, with a denoising diffusion probabilistic model (DDPM) to generate natural-looking results. To seamlessly fuse the edited region with the unchanged parts of the image, we spatially blend noised versions of the input image with the local text-guided diffusion latent at a progression of noise levels. In addition, we show that adding augmentations to the diffusion process mitigates adversarial results. We compare against several baselines and related methods, both qualitatively and quantitatively, and show that our method outperforms these solutions in terms of overall realism, ability to preserve the background and matching the text. Finally, we show several text-driven editing applications, including adding a new object to an image, removing/replacing/altering existing objects, background replacement, and image extrapolation. Code is available at: https://omriavrahami.com/blended-diffusion-page/

Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech

Abstract: Recently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by gradually transforming noise predicted by encoder and aligned with text input by means of Monotonic Alignment Search. The framework of stochastic differential equations helps us to generalize conventional diffusion probabilistic models to the case of reconstructing data from noise with different parameters and allows to make this reconstruction flexible by explicitly controlling trade-off between sound quality and inference speed. Subjective human evaluation shows that Grad-TTS is competitive with state-of-the-art text-to-speech approaches in terms of Mean Opinion Score. We will make the code publicly available shortly.

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Abstract: Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e.g., mel-spectrogram) given a music score. Previous singing acoustic models adopt a simple loss (e.g., L1 and L2) or generative adversarial network (GAN) to reconstruct the acoustic features, while they suffer from over-smoothing and unstable training issues respectively, which hinder the naturalness of synthesized singing. In this work, we propose DiffSinger, an acoustic model for SVS based on the diffusion probabilistic model. DiffSinger is a parameterized Markov chain that iteratively converts the noise into mel-spectrogram conditioned on the music score. By implicitly optimizing variational bound, DiffSinger can be stably trained and generate realistic outputs. To further improve the voice quality and speed up inference, we introduce a shallow diffusion mechanism to make better use of the prior knowledge learned by the simple loss. Specifically, DiffSinger starts generation at a shallow step smaller than the total number of diffusion steps, according to the intersection of the diffusion trajectories of the ground-truth mel-spectrogram and the one predicted by a simple mel-spectrogram decoder. Besides, we propose boundary prediction methods to locate the intersection and determine the shallow step adaptively. The evaluations conducted on a Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work. Extensional experiments also prove the generalization of our methods on text-to-speech task (DiffSpeech). Audio samples: https://diffsinger.github.io. Codes: https://github.com/MoonInTheRiver/DiffSinger. The old title of this work: "Diffsinger: Diffusion acoustic model for singing voice synthesis".

Torsional Diffusion for Molecular Conformer Generation

Abstract: Molecular conformer generation is a fundamental task in computational chemistry. Several machine learning approaches have been developed, but none have outperformed state-of-the-art cheminformatics methods. We propose torsional diffusion, a novel diffusion framework that operates on the space of torsion angles via a diffusion process on the hypertorus and an extrinsic-to-intrinsic score model. On a standard benchmark of drug-like molecules, torsional diffusion generates superior conformer ensembles compared to machine learning and cheminformatics methods in terms of both RMSD and chemical properties, and is orders of magnitude faster than previous diffusion-based models. Moreover, our model provides exact likelihoods, which we employ to build the first generalizable Boltzmann generator. Code is available at https://github.com/gcorso/torsional-diffusion.

Diffusion-based Molecule Generation with Informative Prior Bridges

Abstract: AI-based molecule generation provides a promising approach to a large area of biomedical sciences and engineering, such as antibody design, hydrolase engineering, or vaccine development. Because the molecules are governed by physical laws, a key challenge is to incorporate prior information into the training procedure to generate high-quality and realistic molecules. We propose a simple and novel approach to steer the training of diffusion-based generative models with physical and statistics prior information. This is achieved by constructing physically informed diffusion bridges, stochastic processes that guarantee to yield a given observation at the fixed terminal time. We develop a Lyapunov function based method to construct and determine bridges, and propose a number of proposals of informative prior bridges for both high-quality molecule generation and uniformity-promoted 3D point cloud generation. With comprehensive experiments, we show that our method provides a powerful approach to the 3D generation task, yielding molecule structures with better quality and stability scores and more uniformly distributed point clouds of high qualities.

CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation

Abstract: The imputation of missing values in time series has many applications in healthcare and finance. While autoregressive models are natural candidates for time series imputation, score-based diffusion models have recently outperformed existing counterparts including autoregressive models in many tasks such as image generation and audio synthesis, and would be promising for time series imputation. In this paper, we propose Conditional Score-based Diffusion models for Imputation (CSDI), a novel time series imputation method that utilizes score-based diffusion models conditioned on observed data. Unlike existing score-based approaches, the conditional diffusion model is explicitly trained for imputation and can exploit correlations between observed values. On healthcare and environmental data, CSDI improves by 40-65% over existing probabilistic imputation methods on popular performance metrics. In addition, deterministic imputation by CSDI reduces the error by 5-20% compared to the state-of-the-art deterministic imputation methods. Furthermore, CSDI can also be applied to time series interpolation and probabilistic forecasting, and is competitive with existing baselines. The code is available at https://github.com/ermongroup/CSDI.

Diffusion Models for Adversarial Purification

Abstract: Adversarial purification refers to a class of defense methods that remove adversarial perturbations using a generative model. These methods do not make assumptions on the form of attack and the classification model, and thus can defend pre-existing classifiers against unseen threats. However, their performance currently falls behind adversarial training methods. In this work, we propose DiffPure that uses diffusion models for adversarial purification: Given an adversarial example, we first diffuse it with a small amount of noise following a forward diffusion process, and then recover the clean image through a reverse generative process. To evaluate our method against strong adaptive attacks in an efficient and scalable way, we propose to use the adjoint method to compute full gradients of the reverse generative process. Extensive experiments on three image datasets including CIFAR-10, ImageNet and CelebA-HQ with three classifier architectures including ResNet, WideResNet and ViT demonstrate that our method achieves the state-of-the-art results, outperforming current adversarial training and adversarial purification methods, often by a large margin. Project page: https://diffpure.github.io.

Summary of all the applications utilizing the diffusion models.

A Collection of Applications with Diffusion Models

Due to the flexibility and strength of diffusion models, they have recently been applied in many real-world applications. In this section, we categorize these applications into seven parts, including computer vision, natural language processing, waveform signal processing, multi-modal learning, molecular graph modeling, time series modeling, and adversarial purification. In each subsection, we first give a brief introduction to each task, and then introduce in detail how diffusion models are utilized to promote the performances. We summarize all the applications that utilize the diffusion models in Table {{table:eb8b40a8-4ba3-45c3-8aa6-a1ec89dc28cb}} .


A box-covering algorithm for fractal scaling in scale-free networks.

Abstract: A random sequential box-covering algorithm recently introduced to measure the fractal dimension in scale-free (SF) networks is investigated. The algorithm contains Monte Carlo sequential steps of choosing the position of the center of each box; thereby, vertices in preassigned boxes can divide subsequent boxes into more than one piece, but divided boxes are counted once. We find that such box-split allowance in the algorithm is a crucial ingredient necessary to obtain the fractal scaling for fractal networks; however, it is inessential for regular lattice and conventional fractal objects embedded in the Euclidean space. Next, the algorithm is viewed from the cluster-growing perspective that boxes are allowed to overlap; thereby, vertices can belong to more than one box. The number of distinct boxes a vertex belongs to is, then, distributed in a heterogeneous manner for SF fractal networks, while it is of Poisson-type for the conventional fractal objects.

How to calculate the fractal dimension of a complex network: the box covering algorithm

Abstract: Covering a network with the minimum possible number of boxes can reveal interesting features for the network structure, especially in terms of self-similar or fractal characteristics. Considerable attention has been recently devoted to this problem, with the finding that many real networks are self-similar fractals. Here we present, compare and study in detail a number of algorithms that we have used in previous papers towards this goal. We show that this problem can be mapped to the well-known graph coloring problem and then we simply can apply well-established algorithms. This seems to be the most efficient method, but we also present two other algorithms based on burning which provide a number of other benefits. We argue that the presented algorithms provide a solution close to optimal and that another algorithm that can significantly improve this result in an efficient way does not exist. We offer to anyone that finds such a method to cover his/her expenses for a 1-week trip to our lab in New York (details in http://jamlab.org).

Betweenness Centrality of Fractal and Non-Fractal Scale-Free Model Networks and Tests on Real Networks

Abstract: We study the betweenness centrality of fractal and non-fractal scale-free network models as well as real networks. We show that the correlation between degree and betweenness centrality $C$ of nodes is much weaker in fractal network models compared to non-fractal models. We also show that nodes of both fractal and non-fractal scale-free networks have power law betweenness centrality distribution $P(C)\sim C^{-\delta}$. We find that for non-fractal scale-free networks $\delta = 2$, and for fractal scale-free networks $\delta = 2-1/d_{B}$, where $d_{B}$ is the dimension of the fractal network. We support these results by explicit calculations on four real networks: pharmaceutical firms (N=6776), yeast (N=1458), WWW (N=2526), and a sample of Internet network at AS level (N=20566), where $N$ is the number of nodes in the largest connected component of a network. We also study the crossover phenomenon from fractal to non-fractal networks upon adding random edges to a fractal network. We show that the crossover length $\ell^{*}$, separating fractal and non-fractal regimes, scales with dimension $d_{B}$ of the network as $p^{-1/d_{B}}$, where $p$ is the density of random edges added to the network. We find that the correlation between degree and betweenness centrality increases with $p$.

Exploring self-similarity of complex cellular networks: The edge-covering method with simulated annealing and log-periodic sampling

Abstract: Song, Havlin and Makse (2005) have recently used a version of the box-counting method, called the node-covering method, to quantify the self-similar properties of 43 cellular networks: the minimal number $N_V$ of boxes of size $\ell$ needed to cover all the nodes of a cellular network was found to scale as the power law $N_V \sim (\ell+1)^{-D_V}$ with a fractal dimension $D_V=3.53\pm0.26$. We propose a new box-counting method based on edge-covering, which outperforms the node-covering approach when applied to strictly self-similar model networks, such as the Sierpinski network. The minimal number $N_E$ of boxes of size $\ell$ in the edge-covering method is obtained with the simulated annealing algorithm. We take into account the possible discrete scale symmetry of networks (artifactual and/or real), which is visualized in terms of log-periodic oscillations in the dependence of the logarithm of $N_E$ as a function of the logarithm of $\ell$. In this way, we are able to remove the bias of the estimator of the fractal dimension, existing for finite networks. With this new methodology, we find that $N_E$ scales with respect to $\ell$ as a power law $N_E \sim \ell^{-D_E}$ with $D_E=2.67\pm0.15$ for the 43 cellular networks previously analyzed by Song, Havlin and Makse (2005). Bootstrap tests suggest that the analyzed cellular networks may have a significant log-periodicity qualifying a discrete hierarchy with a scaling ratio close to 2. In sum, we propose that our method of edge-covering with simulated annealing and log-periodic sampling minimizes the significant bias in the determination of fractal dimensions in log-log regressions.

Optimal box-covering algorithm for fractal dimension of complex networks

Abstract: The self-similarity of complex networks is typically investigated through computational algorithms, the primary task of which is to cover the structure with a minimal number of boxes. Here we introduce a box-covering algorithm that outperforms previous ones in most cases. For the two benchmark cases tested, namely, the E. coli and the World Wide Web (WWW) networks, our results show that the improvement can be rather substantial, reaching up to 15% in the case of the WWW network.

Modelling the Self-similarity in Complex Networks Based on Coulomb's Law

Abstract: Recently, self-similarity of complex networks have attracted much attention. Fractal dimension of complex network is an open issue. Hub repulsion plays an important role in fractal topologies. This paper models the repulsion among the nodes in the complex networks in calculation of the fractal dimension of the networks. The Coulomb's law is adopted to represent the repulse between two nodes of the network quantitatively. A new method to calculate the fractal dimension of complex networks is proposed. The Sierpinski triangle network and some real complex networks are investigated. The results are illustrated to show that the new model of self-similarity of complex networks is reasonable and efficient.

Fractality of Massive Graphs: Scalable Analysis with Sketch-Based Box-Covering Algorithm

Abstract: Analysis and modeling of networked objects are fundamental pieces of modern data mining. Most real-world networks, from biological to social ones, are known to have common structural properties. These properties allow us to model the growth processes of networks and to develop useful algorithms. One remarkable example is the fractality of networks, which suggests the self-similar organization of global network structure. To determine the fractality of a network, we need to solve the so-called box-covering problem, where preceding algorithms are not feasible for large-scale networks. The lack of an efficient algorithm prevents us from investigating the fractal nature of large-scale networks. To overcome this issue, we propose a new box-covering algorithm based on recently emerging sketching techniques. We theoretically show that it works in near-linear time with a guarantee of solution accuracy. In experiments, we have confirmed that the algorithm enables us to study the fractality of million-scale networks for the first time. We have observed that its outputs are sufficiently accurate and that its time and space requirements are orders of magnitude smaller than those of previous algorithms.

The box-covering algorithms. The starred algorithms are not yet available in our repository.

The box-covering of a network is known to be NP-hard {{cite:29925a4}}. Hence to have an algorithm of practically acceptable time complexity, one must use approximating methods. As it is usual for semi-empirical methods, there are a good number of different proposals for the task, see Table {{table:f35bfcbf-641c-441f-9e61-fb0a3e79122d}} . To give some intuitive summary, we can say that most algorithms follow a greedy strategy where the distinction between algorithms is drawn by the actual greedy decision method. However, we present a number of approaches beyond the greedy paradigm.


Cuckoo Search via Levy Flights

Abstract: In this paper, we intend to formulate a new metaheuristic algorithm, called Cuckoo Search (CS), for solving optimization problems. This algorithm is based on the obligate brood parasitic behaviour of some cuckoo species in combination with the Levy flight behaviour of some birds and fruit flies. We validate the proposed algorithm against test functions and then compare its performance with those of genetic algorithms and particle swarm optimization. Finally, we discuss the implication of the results and suggestion for further research.

Model-Free Optimization Using Eagle Perching Optimizer

Abstract: The paper proposes a novel nature-inspired technique of optimization. It mimics the perching nature of eagles and uses mathematical formulations to introduce a new addition to metaheuristic algorithms. The nature of the proposed algorithm is based on exploration and exploitation. The proposed algorithm is developed into two versions with some modifications. In the first phase, it undergoes a rigorous analysis to find out their performance. In the second phase it is benchmarked using ten functions of two categories; uni-modal functions and multi-modal functions. In the third phase, we conducted a detailed analysis of the algorithm by exploiting its controlling units or variables. In the fourth and last phase, we consider real world optimization problems with constraints. Both versions of the algorithm show an appreciable performance, but analysis puts more weight to the modified version. The competitive analysis shows that the proposed algorithm outperforms the other tested metaheuristic algorithms. The proposed method has better robustness and computational efficiency.

Eagle Strategy Using L\'evy Walk and Firefly Algorithms For Stochastic Optimization

Abstract: Most global optimization problems are nonlinear and thus difficult to solve, and they become even more challenging when uncertainties are present in objective functions and constraints. This paper provides a new two-stage hybrid search method, called Eagle Strategy, for stochastic optimization. This strategy intends to combine the random search using L\'evy walk with the firefly algorithm in an iterative manner. Numerical studies and results suggest that the proposed Eagle Strategy is very efficient for stochastic optimization. Finally practical implications and potential topics for further research will be discussed.

New Hoopoe Heuristic Optimization

Abstract: Most optimization problems in real life applications are often highly nonlinear. Local optimization algorithms do not give the desired performance. So, only global optimization algorithms should be used to obtain optimal solutions. This paper introduces a new nature-inspired metaheuristic optimization algorithm, called Hoopoe Heuristic (HH). In this paper, we will study HH and validate it against some test functions. Investigations show that it is very promising and could be seen as an optimization of the powerful algorithm of cuckoo search. Finally, we discuss the features of Hoopoe Heuristic and propose topics for further studies.

Algorithms based on birds behavior.

Mating and breeding is found in cuckoo bird which is considered a parasite where it found more promises nest to lay eggs ( {{cite:3f2aadd}}) or mature cuckoos and eggs which try to survives in the population ( {{cite:feb6a91}}). Other algorithm use bird mating strategies to increase opportunity of breeding ( {{cite:781fa37}}); per example, laying chicken hatching eggs ( {{cite:b78b6e5}}) or during breeding, See-See partridge chicks follow their mother to reach a safe place ( {{cite:eb33527}}) and attraction of females by the satin birds males ( {{cite:7414dad}}). All algorithms based on birds behavior are shown in Table  {{table:8e4dfbf3-86cf-4d41-932c-2dcf4161b308}}


Revisiting the Negative Data of Distantly Supervised Relation Extraction

Abstract: Distantly supervision automatically generates plenty of training samples for relation extraction. However, it also incurs two major problems: noisy labels and imbalanced training data. Previous works focus more on reducing wrongly labeled relations (false positives) while few explore the missing relations that are caused by incompleteness of knowledge base (false negatives). Furthermore, the quantity of negative labels overwhelmingly surpasses the positive ones in previous problem formulations. In this paper, we first provide a thorough analysis of the above challenges caused by negative data. Next, we formulate the problem of relation extraction into as a positive unlabeled learning task to alleviate false negative problem. Thirdly, we propose a pipeline approach, dubbed ReRe, that first performs sentence classification with relational labels and then extracts the subjects/objects. Experimental results show that the proposed method consistently outperforms existing approaches and remains excellent performance even learned with a large quantity of false positive samples. Source code is available online at https://github.com/redreamality/RERE-relation-extraction.

A Hierarchical Framework for Relation Extraction with Reinforcement Learning

Abstract: Most existing methods determine relation types only after all the entities have been recognized, thus the interaction between relation types and entity mentions is not fully modeled. This paper presents a novel paradigm to deal with relation extraction by regarding the related entities as the arguments of a relation. We apply a hierarchical reinforcement learning (HRL) framework in this paradigm to enhance the interaction between entity mentions and relation types. The whole extraction process is decomposed into a hierarchy of two-level RL policies for relation detection and entity extraction respectively, so that it is more feasible and natural to deal with overlapping relations. Our model was evaluated on public datasets collected via distant supervision, and results show that it gains better performance than existing methods and is more powerful for extracting overlapping relations1.

The statistic of sensitive instances on different datasets, where an instance is sensitive if different extraction orders produces different extraction results with the same model.

However, we find that different extraction orders highly affect the extraction results.
For example, in the A instance in Fig. {{figure:9a960b4a-a241-4381-aa56-7f1c0e9d2ac0}} , if we first extract the Bassoon Concerto in B-flat major, it will be difficult for succeeding element extraction because the entity is long-tail. Instead, if we extract Mozart first, it would be much easier to extract the concerto because Mozart appears frequently in the corpus and he composed lots of concerto.
According to our observation experiments conducted on several popular relation extraction and event extraction datasets as listed in Table {{table:fa372f0c-7b56-45e2-8b2e-f88d1e98dfd6}} , a significant proportion of sentences are sensitive to the extraction order, i.e., different extraction order produces different extraction results.
The ratio of sensitive sentences increases dramatically with the complexity of the task on different datasets.


Explaining Relationships Between Scientific Documents

Abstract: We address the task of explaining relationships between two scientific documents using natural language text. This task requires modeling the complex content of long technical documents, deducing a relationship between these documents, and expressing the details of that relationship in text. In addition to the theoretical interest of this task, successful solutions can help improve researcher efficiency in search and review. In this paper we establish a dataset of 622K examples from 154K documents. We pretrain a large language model to serve as the foundation for autoregressive approaches to the task. We explore the impact of taking different views on the two documents, including the use of dense representations extracted with scientific IE systems. We provide extensive automatic and human evaluations which show the promise of such models, but make clear challenges for future work.

Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation

Abstract: Conventional neural networks show a powerful framework for background subtraction in video acquired by static cameras. Indeed, the well-known SOBS method and its variants based on neural networks were the leader methods on the largescale CDnet 2012 dataset during a long time. Recently, convolutional neural networks which belong to deep learning methods were employed with success for background initialization, foreground detection and deep learned features. Currently, the top current background subtraction methods in CDnet 2014 are based on deep neural networks with a large gap of performance in comparison on the conventional unsupervised approaches based on multi-features or multi-cues strategies. Furthermore, a huge amount of papers was published since 2016 when Braham and Van Droogenbroeck published their first work on CNN applied to background subtraction providing a regular gain of performance. In this context, we provide the first review of deep neural network concepts in background subtraction for novices and experts in order to analyze this success and to provide further directions. For this, we first surveyed the methods used background initialization, background subtraction and deep learned features. Then, we discuss the adequacy of deep neural networks for background subtraction. Finally, experimental results are presented on the CDnet 2014 dataset.

A summary of the problem formulations of the prior works on neural network-based related work generation. “Context” refers to those sentences or paragraphs around the target citation sentences.

In addition to the different understanding of the task definition, each work may have different scopes of the task, such as only addressing part of the task steps or only considering part of the input papers. In this section, we compare each prior work's specific task definition, and the input and output of their systems. Table {{table:3a228770-e683-4b8c-a4c7-f2898dad92ce}}  summarizes the inputs to the extractive systems, and Table {{table:d3d6cf97-4378-4648-89dc-69d81542cfba}}  summarizes the inputs and outputs to the abstractive systems.


Feature Pyramid Networks for Object Detection

Abstract: Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But pyramid representations have been avoided in recent object detectors that are based on deep convolutional networks, partially because they are slow to compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles, surpassing all existing single-model entries including those from the COCO 2016 challenge winners. In addition, our method can run at 5 FPS on a GPU and thus is a practical and accurate solution to multi-scale object detection. Code will be made publicly available.

R-FCN: Object Detection via Region-based Fully Convolutional Networks

Abstract: We present region-based, fully convolutional networks for accurate and efficient object detection. In contrast to previous region-based detectors such as Fast/Faster R-CNN that apply a costly per-region subnetwork hundreds of times, our region-based detector is fully convolutional with almost all computation shared on the entire image. To achieve this goal, we propose position-sensitive score maps to address a dilemma between translation-invariance in image classification and translation-variance in object detection. Our method can thus naturally adopt fully convolutional image classifier backbones, such as the latest Residual Networks (ResNets), for object detection. We show competitive results on the PASCAL VOC datasets (e.g., 83.6% mAP on the 2007 set) with the 101-layer ResNet. Meanwhile, our result is achieved at a test-time speed of 170ms per image, 2.5-20 times faster than the Faster R-CNN counterpart. Code is made publicly available at: https://github.com/daijifeng001/r-fcn.

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Abstract: State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available

Model performance and speed. The first number in the AUC column denotes the result on the small test set, the second number - on the test set of data for the last two days (819 images for training, 222 images for testing).

In Table {{table:62b630df-67a5-4383-8ff2-e44c59608a99}}  we list our results and measurements.


Adversarial Reprogramming of Neural Networks

Abstract: Deep neural networks are susceptible to \emph{adversarial} attacks. In computer vision, well-crafted perturbations to images can cause neural networks to make mistakes such as confusing a cat with a computer. Previous adversarial attacks have been designed to degrade performance of models or cause machine learning models to produce specific outputs chosen ahead of time by the attacker. We introduce attacks that instead {\em reprogram} the target model to perform a task chosen by the attacker---without the attacker needing to specify or compute the desired output for each test-time input. This attack finds a single adversarial perturbation, that can be added to all test-time inputs to a machine learning model in order to cause the model to perform a task chosen by the adversary---even if the model was not trained to do this task. These perturbations can thus be considered a program for the new task. We demonstrate adversarial reprogramming on six ImageNet classification models, repurposing these models to perform a counting task, as well as classification tasks: classification of MNIST and CIFAR-10 examples presented as inputs to the ImageNet model.

Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources

Abstract: Current transfer learning methods are mainly based on finetuning a pretrained model with target-domain data. Motivated by the techniques from adversarial machine learning (ML) that are capable of manipulating the model prediction via data perturbations, in this paper we propose a novel approach, black-box adversarial reprogramming (BAR), that repurposes a well-trained black-box ML model (e.g., a prediction API or a proprietary software) for solving different ML tasks, especially in the scenario with scarce data and constrained resources. The rationale lies in exploiting high-performance but unknown ML models to gain learning capability for transfer learning. Using zeroth order optimization and multi-label mapping techniques, BAR can reprogram a black-box ML model solely based on its input-output responses without knowing the model architecture or changing any parameter. More importantly, in the limited medical data setting, on autism spectrum disorder classification, diabetic retinopathy detection, and melanoma detection tasks, BAR outperforms state-of-the-art methods and yields comparable performance to the vanilla adversarial reprogramming method requiring complete knowledge of the target ML model. BAR also outperforms baseline transfer learning approaches by a significant margin, demonstrating cost-effective means and new insights for transfer learning.

Reprogramming Language Models for Molecular Representation Learning

Abstract: Recent advancements in transfer learning have made it a promising approach for domain adaptation via transfer of learned representations. This is especially when relevant when alternate tasks have limited samples of well-defined and labeled data, which is common in the molecule data domain. This makes transfer learning an ideal approach to solve molecular learning tasks. While Adversarial reprogramming has proven to be a successful method to repurpose neural networks for alternate tasks, most works consider source and alternate tasks within the same domain. In this work, we propose a new algorithm, Representation Reprogramming via Dictionary Learning (R2DL), for adversarially reprogramming pretrained language models for molecular learning tasks, motivated by leveraging learned representations in massive state of the art language models. The adversarial program learns a linear transformation between a dense source model input space (language data) and a sparse target model input space (e.g., chemical and biological molecule data) using a k-SVD solver to approximate a sparse representation of the encoded data, via dictionary learning. R2DL achieves the baseline established by state of the art toxicity prediction models trained on domain-specific data and outperforms the baseline in a limited training-data setting, thereby establishing avenues for domain-agnostic transfer learning for tasks with molecule data.

WARP: Word-level Adversarial ReProgramming

Abstract: Transfer learning from pretrained language models recently became the dominant approach for solving many NLP tasks. A common approach to transfer learning for multiple tasks that maximize parameter sharing trains one or more task-specific layers on top of the language model. In this paper, we present an alternative approach based on adversarial reprogramming, which extends earlier work on automatic prompt generation. Adversarial reprogramming attempts to learn task-specific word embeddings that, when concatenated to the input text, instruct the language model to solve the specified task. Using up to 25K trainable parameters per task, this approach outperforms all existing methods with up to 25M trainable parameters on the public leaderboard of the GLUE benchmark. Our method, initialized with task-specific human-readable prompts, also works in a few-shot setting, outperforming GPT-3 on two SuperGLUE tasks with just 32 training samples.

Voice2Series: Reprogramming Acoustic Models for Time Series Classification

Abstract: Learning to classify time series with limited data is a practical yet challenging problem. Current methods are primarily based on hand-designed feature extraction rules or domain-specific data augmentation. Motivated by the advances in deep speech processing models and the fact that voice data are univariate temporal signals, in this paper, we propose Voice2Series (V2S), a novel end-to-end approach that reprograms acoustic models for time series classification, through input transformation learning and output label mapping. Leveraging the representation learning power of a large-scale pre-trained speech processing model, on 30 different time series tasks we show that V2S performs competitive results on 19 time series classification tasks. We further provide a theoretical justification of V2S by proving its population risk is upper bounded by the source risk and a Wasserstein distance accounting for feature alignment via reprogramming. Our results offer new and effective means to time series classification.

Cross-modal Adversarial Reprogramming

Abstract: With the abundance of large-scale deep learning models, it has become possible to repurpose pre-trained networks for new tasks. Recent works on adversarial reprogramming have shown that it is possible to repurpose neural networks for alternate tasks without modifying the network architecture or parameters. However these works only consider original and target tasks within the same data domain. In this work, we broaden the scope of adversarial reprogramming beyond the data modality of the original task. We analyze the feasibility of adversarially repurposing image classification neural networks for Natural Language Processing (NLP) and other sequence classification tasks. We design an efficient adversarial program that maps a sequence of discrete tokens into an image which can be classified to the desired class by an image classification model. We demonstrate that by using highly efficient adversarial programs, we can reprogram image classifiers to achieve competitive performance on a variety of text and sequence classification benchmarks without retraining the network.

Summary of model reprogramming use cases. LSTM means long short-term memory, CNN means convolutional neural network, API means application programming interface, and SOTA means state of the art. BERT stands for bidirectional encoder representations from transformers. GLUE stands for the general language understanding evaluation benchmark. GAN stands for generative adversarial network. CA stands for cellular automata. ViT stands for vision transformer.We also maintain a list of model reprogramming studies at https://github.com/IBM/model-reprogramming.

Model reprogramming has shown success and improved performance for resource-efficient cross-domain machine learning on a wide range of data domains, pre-trained source models, and machine learning tasks.
Table {{table:c2e137d3-2e01-47e0-949e-ee4b403189ed}}  summarizes existing studies on model reprogramming. In what follows we highlight two representative use cases for each data format (continuous or discrete) featuring improved task performance and resource efficiency.


The Video Head Impulse Test (vHIT) Detects Vertical Semicircular Canal Dysfunction

Abstract: Background The video head impulse test (vHIT) is a useful clinical tool to detect semicircular canal dysfunction. However vHIT has hitherto been limited to measurement of horizontal canals, while scleral search coils have been the only accepted method to measure head impulses in vertical canals. The goal of this study was to determine whether vHIT can detect vertical semicircular canal dysfunction as identified by scleral search coil recordings. Methods Small unpredictable head rotations were delivered by hand diagonally in the plane of the vertical semicircular canals while gaze was directed along the same plane. The planes were oriented along the left-anterior-right-posterior (LARP) canals and right-anterior-left-posterior (RALP) canals. Eye movements were recorded simultaneously in 2D with vHIT (250 Hz) and in 3D with search coils (1000 Hz). Twelve patients with unilateral, bilateral and individual semicircular canal dysfunction were tested and compared to seven normal subjects. Results Simultaneous video and search coil recordings were closely comparable. Mean VOR gain difference measured with vHIT and search coils was 0.05 (SD = 0.14) for the LARP plane and −0.04 (SD = 0.14) for the RALP plane. The coefficient of determination R2 was 0.98 for the LARP plane and 0.98 for the RALP plane and the results of the two methods were not significantly different. vHIT and search coil measures displayed comparable patterns of covert and overt catch-up saccades. Conclusions vHIT detects dysfunction of individual vertical semicircular canals in vestibular patients as accurately as scleral search coils. Unlike search coils, vHIT is non-invasive, easy to use and hence practical in clinics.

Eye Movement Signal Classification for Developing Human-Computer Interface Using Electrooculogram

Abstract: Human-computer interfaces (HCI) allow people to control electronic devices, such as computers, mouses, wheelchairs, and keyboards, by bypassing the biochannel without using motor nervous system signals. These signals permit communication between people and electronic-controllable devices. This communication is due to HCI, which facilitates lives of paralyzed patients who do not have any problems with their cognitive functioning. The major plan of this study is to test out the feasibility of nine states of HCI by using modern techniques to overcome the problem faced by the paralyzed. Analog Digital Instrument T26 with a five-electrode system was used in this method. Voluntarily twenty subjects participated in this study. The extracted signals were preprocessed by applying notch filter with a range of 50 Hz to remove the external interferences; the features were extracted by applying convolution theorem. Afterwards, extracted features were classified using Elman and distributed time delay neural network. Average classification accuracy with 90.82% and 90.56% was achieved using two network models. The accuracy of the classifier was analyzed by single-trial analysis and performances of the classifier were observed using bit transfer rate (BTR) for twenty subjects to check the feasibility of designing the HCI. The achieved results showed that the ERNN model has a greater potential to classify, identify, and recognize the EOG signal compared with distributed time delay network for most of the subjects. The control signal generated by classifiers was applied as control signals to navigate the assistive devices such as mouse, keyboard, and wheelchair activities for disabled people.

Sampling Rates for Scleral Search Coil (SSC) and EOG Methods

This statement may have been true when {{cite:75c1d58}} made it in 1993. EyeLink devices at that time had sampling rates of 250 Hz. But as of today, this statement no longer holds. As is evident in Table {{table:294c73b1-4bdd-4b95-bb11-0a09ccfdf154}} , for the search coil studies, the lowest sampling rate was 200 Hz and the highest was 1000 Hz. The median sampling rate for the search coil studies was 1000 Hz. For EOG, the lowest rate was 100 Hz and the highest rate was 10,000 Hz. The median sampling rate for EOG studies was 762 Hz. Only one research group employed a 10,000 Hz sampling rate for EOG {{cite:b68a8e8}}, {{cite:873329d}}. All of the other papers reported sampling rates of 1024 or less. Looking at the sampling rates in Table {{table:294c73b1-4bdd-4b95-bb11-0a09ccfdf154}} , and considering that the EyeLink systems now use sampling rates of 500 Hz, 1000 Hz or 2000 Hz, there is no basis for saying that the EyeLink sample rates are low at the current time.


Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

Abstract: First-person vision is gaining interest as it offers a unique viewpoint on people's interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict nonscripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labeled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens. Dataset and Project page: this http URL

MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration

Abstract: Multimodal video-audio-text understanding and generation can benefit from datasets that are narrow but rich. The narrowness allows bite-sized challenges that the research community can make progress on. The richness ensures we are making progress along the core challenges. To this end, we present a large-scale video-audio-text dataset MUGEN, collected using the open-sourced platform game CoinRun [11]. We made substantial modifications to make the game richer by introducing audio and enabling new interactions. We trained RL agents with different objectives to navigate the game and interact with 13 objects and characters. This allows us to automatically extract a large collection of diverse videos and associated audio. We sample 375K video clips (3.2s each) and collect text descriptions from human annotators. Each video has additional annotations that are extracted automatically from the game engine, such as accurate semantic maps for each frame and templated textual descriptions. Altogether, MUGEN can help progress research in many tasks in multimodal understanding and generation. We benchmark representative approaches on tasks involving video-audio-text retrieval and generation. Our dataset and code are released at: https://mugen-org.github.io/.

The statistics of our used datsets to evaluate TVC.

Datasets.
As a new task, we consider diverse video scenes with natural instructions for TVC. Kitchen {{cite:96b0792}} records 22K egocentric videos about kitchen activity, which have different lengths (4-16 frames) with narrations. Flintstones {{cite:eaad892}} contains 25K animation videos (15 frames) from The Flintstones, where each video description includes the characters and their behavior. MUGEN {{cite:4f79f71}} is built from agents playing CoinRun {{cite:5d72a76}}, which consists of 375K gaming videos (16 frames) with detailed text annotations. All videos in these three datasets are resized into 128x128. An overview is shown in Table {{table:169b315a-6895-4dba-81ba-faecc9340fd3}}  and Fig. {{figure:157bed20-271f-4b84-b7f3-69c0696cdd35}} (b). Since MMVG can unify various tasks, we also evaluate video generation/prediction on widely-used UCF-101 {{cite:8358991}} and BAIR {{cite:52bf762}}, video infilling on UCF-101 following RaMViD {{cite:1b623e6}}, and text-to-video generation on MSRVTT {{cite:c056a8d}}.


SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

Abstract: We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from different layers, and thus combining both local attention and global attention to render powerful representations. We show that this simple and lightweight design is the key to efficient segmentation on Transformers. We scale our approach up to obtain a series of models from SegFormer-B0 to SegFormer-B5, reaching significantly better performance and efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters, being 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on Cityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C. Code will be released at: github.com/NVlabs/SegFormer.

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Abstract: This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at https://github.com/microsoft/Swin-Transformer.

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Abstract: DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10$\times$ less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach. Code shall be released.

State-of-the-art vision transformer model summary for a batch size of 1 and profiling on a NVIDIA RTX A5000 GPU with clocks locked to 1005 MHz. SS = semantic segmentation. OD = object detection. Accuracy metrics: mIoU for the ADE20K {{cite:6d8b8e1}} and Cityscapes {{cite:04d3f8c}} datasets (SS) and AP, with IoU from 0.5 to 0.95 in increments of 0.05 for the COCO-2017 dataset {{cite:6125946}} (OD).

Resiliency of Pretrained Semantic Segmentation Models

We further explore the resiliency of SegFormer across image size (with different datasets) by running our experiments for the SegFormer B2 model trained with the CityScapes dataset {{cite:04d3f8c}}, which uses larger image sizes (1024 by 2048) and thus has longer inference execution times than the ADE20K dataset. As shown in Table {{table:c04aa2b9-1138-4fe6-8d4a-c098d546c96d}} , the SegFormer Cityscapes model has higher accuracy on the Cityscapes validation dataset (0.8098) compared to model trained on and validated with the ADE20K dataset (0.4651). We examine the resiliency of these models using all the datasets that the model papers reported accuracy on: Swin Transformer published models with the ADE20K dataset while SegFormer published models for the ADE20K and CityScapes datasets. Model layers are denoted with the labels used in Figure {{figure:59db8b4b-edc9-48fa-ac50-600d13f4b4f8}} . Note that fpn_bottleneck_Conv2D refers to a large convolution in the Swin decoder, similar to SegFormer's Conv2DFuse. Experiments were run on a NVIDIA RTX A5000 GPU, with clocks locked to 1005 MHz.


FoodX-251: A Dataset for Fine-grained Food Classification

Abstract: Food classification is a challenging problem due to the large number of categories, high visual similarity between different foods, as well as the lack of datasets for training state-of-the-art deep models. Solving this problem will require advances in both computer vision models as well as datasets for evaluating these models. In this paper we focus on the second aspect and introduce FoodX-251, a dataset of 251 fine-grained food categories with 158k images collected from the web. We use 118k images as a training set and provide human verified labels for 40k images that can be used for validation and testing. In this work, we outline the procedure of creating this dataset and provide relevant baselines with deep learning models. The FoodX-251 dataset has been used for organizing iFood-2019 challenge in the Fine-Grained Visual Categorization workshop (FGVC6 at CVPR 2019) and is available for download.

ChineseFoodNet: A large-scale Image Dataset for Chinese Food Recognition

Abstract: In this paper, we introduce a new and challenging large-scale food image dataset called "ChineseFoodNet", which aims to automatically recognizing pictured Chinese dishes. Most of the existing food image datasets collected food images either from recipe pictures or selfie. In our dataset, images of each food category of our dataset consists of not only web recipe and menu pictures but photos taken from real dishes, recipe and menu as well. ChineseFoodNet contains over 180,000 food photos of 208 categories, with each category covering a large variations in presentations of same Chinese food. We present our efforts to build this large-scale image dataset, including food category selection, data collection, and data clean and label, in particular how to use machine learning methods to reduce manual labeling work that is an expensive process. We share a detailed benchmark of several state-of-the-art deep convolutional neural networks (CNNs) on ChineseFoodNet. We further propose a novel two-step data fusion approach referred as "TastyNet", which combines prediction results from different CNNs with voting method. Our proposed approach achieves top-1 accuracies of 81.43% on the validation set and 81.55% on the test set, respectively. The latest dataset is public available for research and can be achieved at https://sites.google.com/view/chinesefoodnet.

MUSEFood: Multi-Sensor-Based Food Volume Estimation on Smartphones

Abstract: Researches have shown that diet recording can help people increase awareness of food intake and improve nutrition management, and thereby maintain a healthier life. Recently, researchers have been working on smartphone-based diet recording methods and applications that help users accomplish two tasks: record what they eat and how much they eat. Although the former task has made great progress through adopting image recognition technology, it is still a challenge to estimate the volume of foods accurately and conveniently. In this paper, we propose a novel method, named MUSEFood, for food volume estimation. MUSEFood uses the camera to capture photos of the food, but unlike existing volume measurement methods, MUSEFood requires neither training images with volume information nor placing a reference object of known size while taking photos. In addition, considering the impact of different containers on the contour shape of foods, MUSEFood uses a multi-task learning framework to improve the accuracy of food segmentation, and uses a differential model applicable for various containers to further reduce the negative impact of container differences on volume estimation accuracy. Furthermore, MUSEFood uses the microphone and the speaker to accurately measure the vertical distance from the camera to the food in a noisy environment, thus scaling the size of food in the image to its actual size. The experiments on real foods indicate that MUSEFood outperforms state-of-the-art approaches, and highly improves the speed of food volume estimation.

List of food datasets

Because of increasing importance of food-aware tasks, many researchers are working hard on the production of food-aware datasets and the development of food recognition. There are three methods to recognize food: food classification, food detection, and food segmentation. Food classification is a task that matches the type of food in an image through a single image, and many public datasets are also released because it is relatively easier than other tasks during the data collection and labeling phase. However, in order to determine a more accurate food intake in the diet management service, it is necessary to pinpoint the real food portion within the image. Therefore, food segmentation are more useful in this service than food classification and food detection which provides information of the food types and the position by expressing in a bounding box. Nevertheless, there are three difficulties in food segmentation. First, as shown in Table {{table:d80b85dc-7d33-4b5c-afe4-709265aaa323}} , released public datasets available for food segmentation are very scarce compared to food classification public datasets, and most datasets are not publicly available even if released. Second, when producing a dataset personally, there are a tremendous variety of food types and it takes a huge labor cost to labeling. Third, food segmentation is still a challenging task because the variations in shape, volume, texture, color, and composition of food are too large.


Genotypic diversity, mutational and environmental robustness.

Relationship between robustness and phenotype variation in large environmental change

No correlation between environment-as-inducer and environment-as-selector.

Selection for difference in phenotypes instead for fitness

Phenotypic robustness can increase phenotypic variability after non-genetic perturbations in gene regulatory circuits

Abstract: Non-genetic perturbations, such as environmental change or developmental noise, can induce novel phenotypes. If an induced phenotype confers a fitness advantage, selection may promote its genetic stabilization. Non-genetic perturbations can thus initiate evolutionary innovation. Genetic variation that is not usually phenotypically visible may play an important role in this process. Populations under stabilizing selection on a phenotype that is robust to mutations can accumulate such variation. After non-genetic perturbations, this variation can become a source of new phenotypes. We here study the relationship between a phenotype's robustness to mutations and a population's potential to generate novel phenotypic variation. To this end, we use a well-studied model of transcriptional regulation circuits. Such circuits are important in many evolutionary innovations. We find that phenotypic robustness promotes phenotypic variability in response to non-genetic perturbations, but not in response to mutation. Our work suggests that non-genetic perturbations may initiate innovation more frequently in mutationally robust gene expression traits.

Emergence of cooperative bistability and robustness of gene regulatory networks

Abstract: Gene regulatory networks (GRNs) are complex systems in which many genes regulate mutually to adapt the cell state to environmental conditions. In addition to function, the GRNs possess several kinds of robustness. This robustness means that systems do not lose their functionality when exposed to disturbances such as mutations or noise, and is widely observed at many levels in living systems. Both function and robustness have been acquired through evolution. In this respect, GRNs utilized in living systems are rare among all possible GRNs. In this study, we explored the fitness landscape of GRNs and investigated how robustness emerged in highly-fit GRNs. We considered a toy model of GRNs with one input gene and one output gene. The difference in the expression level of the output gene between two input states, “on” and “off”, was considered as fitness. Thus, the determination of the fitness of a GRN was based on how sensitively it responded to the input. We employed the multicanonical Monte Carlo method, which can sample GRNs randomly in a wide range of fitness levels, and classified the GRNs according to their fitness. As a result, the following properties were found: (1) Highly-fit GRNs exhibited bistability for intermediate input between “on” and “off”. This means that such GRNs responded to two input states by using different fixed points of dynamics. This bistability emerges necessarily as fitness increases. (2) These highly-fit GRNs were robust against noise because of their bistability. In other words, noise robustness is a byproduct of high fitness. (3) GRNs that were robust against mutations were not extremely rare among the highly-fit GRNs. This implies that mutational robustness is readily acquired through the evolutionary process. These properties are universal irrespective of the evolutionary pathway, because the results do not rely on evolutionary simulation.

Gene Regulatory Network Models and their results

The Wagner model and its variants have been employed to investigate evolution of robustness {{cite:9032ab8}}, {{cite:8c6a6fd}}, {{cite:67a7fea}}, evolutionary capacitance {{cite:6835455}}, genetic assimilation {{cite:8555b04}},  developmental bias {{cite:56ca8a9}}, relationship between robustness and non-genetic perturbations {{cite:8c6a6fd}}, evolution of novel phenotypes {{cite:6262449}}, {{cite:67a7fea}}, evolution of modularity {{cite:72ae1a5}}, selection for non-stationarity {{cite:4dba04c}}, emergence of bistability {{cite:41fd856}} and how phenotypic plasticity impacts evolution {{cite:317e9a5}}, {{cite:92d0ebe}}. We briefly summarized the key results  alongside their limitations of these works in Table {{table:3901c187-e07d-4b59-88d5-6091de3a5d07}} .  We stress that these limitations are not shortcomings of their models, but rather due to the specific goals of their works. As can be seen in these works, the Wagner model seamlessly incorporates not only the  biological principles of natural selection and genetics as well as the developmental process which interweaves genetic and environmental information into the phenotype {{cite:d63d986}}. By observing differences in adult phenotypes in response to different environmental cues, we see that the Wagner model can exhibit phenotypic plasticity through the developmental process. Due to these advantages over traditional evolutionary models, we believe that the Wagner model is a strong candidate for studying plasticity-led evolution. We will now discuss how well the Wagner model and its variants describe plasticity-led evolution.


Analyzing and Improving the Image Quality of StyleGAN

Abstract: The style-based GAN architecture (StyleGAN) yields state-of-the-art results in data-driven unconditional generative image modeling. We expose and analyze several of its characteristic artifacts, and propose changes in both model architecture and training methods to address them. In particular, we redesign the generator normalization, revisit progressive growing, and regularize the generator to encourage good conditioning in the mapping from latent codes to images. In addition to improving image quality, this path length regularizer yields the additional benefit that the generator becomes significantly easier to invert. This makes it possible to reliably attribute a generated image to a particular network. We furthermore visualize how well the generator utilizes its output resolution, and identify a capacity problem, motivating us to train larger models for additional quality improvements. Overall, our improved model redefines the state of the art in unconditional image modeling, both in terms of existing distribution quality metrics as well as perceived image quality.

Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation

Abstract: We present a generic image-to-image translation framework, pixel2style2pixel (pSp). Our pSp framework is based on a novel encoder network that directly generates a series of style vectors which are fed into a pretrained StyleGAN generator, forming the extended $\mathcal{W} + $ latent space. We first show that our encoder can directly embed real images into $\mathcal{W} + $, with no additional optimization. Next, we propose utilizing our encoder to directly solve image-to-image translation tasks, defining them as encoding problems from some input domain into the latent domain. By deviating from the standard "invert first, edit later" methodology used with previous StyleGAN encoders, our approach can handle a variety of tasks even when the input image is not represented in the StyleGAN domain. We show that solving translation tasks through StyleGAN significantly simplifies the training process, as no adversary is required, has better support for solving tasks without pixel-to-pixel correspondence, and inherently supports multi-modal synthesis via the resampling of styles. Finally, we demonstrate the potential of our framework on a variety of facial image-to-image translation tasks, even when compared to state-of-the-art solutions designed specifically for a single task, and further show that it can be extended beyond the human facial domain. Code is available at https://github.com/eladrich/pixel2style2pixel.

Designing an encoder for StyleGAN image manipulation

Abstract: Recently, there has been a surge of diverse methods for performing image editing by employing pre-trained unconditional generators. Applying these methods on real images, however, remains a challenge, as it necessarily requires the inversion of the images into their latent space. To successfully invert a real image, one needs to find a latent code that reconstructs the input image accurately, and more importantly, allows for its meaningful manipulation. In this paper, we carefully study the latent space of StyleGAN, the state-of-the-art unconditional generator. We identify and analyze the existence of a distortion-editability tradeoff and a distortion-perception tradeoff within the StyleGAN latent space. We then suggest two principles for designing encoders in a manner that allows one to control the proximity of the inversions to regions that StyleGAN was originally trained on. We present an encoder based on our two principles that is specifically designed for facilitating editing on real images by balancing these tradeoffs. By evaluating its performance qualitatively and quantitatively on numerous challenging domains, including cars and horses, we show that our inversion method, followed by common editing techniques, achieves superior real-image editing quality, with only a small reconstruction accuracy drop.

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement

Abstract: Recently, the power of unconditional image synthesis has significantly advanced through the use of Generative Adversarial Networks (GANs). The task of inverting an image into its corresponding latent code of the trained GAN is of utmost importance as it allows for the manipulation of real images, leveraging the rich semantics learned by the network. Recognizing the limitations of current inversion approaches, in this work we present a novel inversion scheme that extends current encoder-based inversion methods by introducing an iterative refinement mechanism. Instead of directly predicting the latent code of a given real image using a single pass, the encoder is tasked with predicting a residual with respect to the current estimate of the inverted latent code in a self-correcting manner. Our residual-based encoder, named ReStyle, attains improved accuracy compared to current state-of-the-art encoder-based methods with a negligible increase in inference time. We analyze the behavior of ReStyle to gain valuable insights into its iterative nature. We then evaluate the performance of our residual encoder and analyze its robustness compared to optimization-based inversion and state-of-the-art encoders.

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

Abstract: Inspired by the ability of StyleGAN to generate highly realistic images in a variety of domains, much recent work has focused on understanding how to use the latent spaces of StyleGAN to manipulate generated and real images. However, discovering semantically meaningful latent manipulations typically involves painstaking human examination of the many degrees of freedom, or an annotated collection of images for each desired manipulation. In this work, we explore leveraging the power of recently introduced Contrastive Language-Image Pre-training (CLIP) models in order to develop a text-based interface for StyleGAN image manipulation that does not require such manual effort. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable text-based manipulation. Finally, we present a method for mapping a text prompts to input-agnostic directions in StyleGAN's style space, enabling interactive text-driven image manipulation. Extensive results and comparisons demonstrate the effectiveness of our approaches.

Training Generative Adversarial Networks with Limited Data

Abstract: Training generative adversarial networks (GAN) using too little data typically leads to discriminator overfitting, causing training to diverge. We propose an adaptive discriminator augmentation mechanism that significantly stabilizes training in limited data regimes. The approach does not require changes to loss functions or network architectures, and is applicable both when training from scratch and when fine-tuning an existing GAN on another dataset. We demonstrate, on several datasets, that good results are now possible using only a few thousand training images, often matching StyleGAN2 results with an order of magnitude fewer images. We expect this to open up new application domains for GANs. We also find that the widely used CIFAR-10 is, in fact, a limited data benchmark, and improve the record FID from 5.59 to 2.42.

Few-shot Image Generation via Cross-domain Correspondence

Abstract: Training generative models, such as GANs, on a target domain containing limited examples (e.g., 10) can easily result in overfitting. In this work, we seek to utilize a large source domain for pretraining and transfer the diversity information from source to target. We propose to preserve the relative similarities and differences between instances in the source via a novel cross-domain distance consistency loss. To further reduce overfitting, we present an anchor-based strategy to encourage different levels of realism over different regions in the latent space. With extensive results in both photorealistic and non-photorealistic domains, we demonstrate qualitatively and quantitatively that our few-shot model automatically discovers correspondences between source and target domains and generates more diverse and realistic images than previous methods.

Models used in our work, their sources and licenses.

Tables {{table:ad177be7-8c12-4c00-824a-02a66c411225}} , {{table:df3e0eb3-d7b2-462b-9d11-a7ca8cba0640}}  provide sources and licenses of the models and datasets we used in our work.