Random Sampling, LSTM Seq2SeqAttn, and GPT-2

LSTM Seq2SeqAttn, BlenderBot, DialoGPT and GPT-3

BlenderBot, DialoGPT, HRED, Transformer/LSTM Seq2Seq

State-of-the-art systems including Plato and DialoGPT

USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation

Abstract: The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for evaluating dialog models. To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces interpretable measures for several desirable properties of dialog.

GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems

Abstract: Automatically evaluating dialogue coherence is a challenging but high-demand ability for developing high-quality open-domain dialogue systems. However, current evaluation metrics consider only surface features or utterance-level semantics, without explicitly considering the fine-grained topic transition dynamics of dialogue flows. Here, we first consider that the graph structure constituted with topics in a dialogue can accurately depict the underlying communication logic, which is a more natural way to produce persuasive metrics. Capitalized on the topic-level dialogue graph, we propose a new evaluation metric GRADE, which stands for Graph-enhanced Representations for Automatic Dialogue Evaluation. Specifically, GRADE incorporates both coarse-grained utterance-level contextualized representations and fine-grained topic-level graph representations to evaluate dialogue coherence. The graph representations are obtained by reasoning over topic-level dialogue graphs enhanced with the evidence from a commonsense graph, including k-hop neighboring representations and hop-attention weights. Experimental results show that our GRADE significantly outperforms other state-of-the-art metrics on measuring diverse dialogue models in terms of the Pearson and Spearman correlations with human judgements. Besides, we release a new large-scale human evaluation benchmark to facilitate future research on automatic metrics.

Designing Precise and Robust Dialogue Response Evaluators

Abstract: Automatic dialogue response evaluator has been proposed as an alternative to automated metrics and human evaluation. However, existing automatic evaluators achieve only moderate correlation with human judgement and they are not robust. In this work, we propose to build a reference-free evaluator and exploit the power of semi-supervised training and pretrained (masked) language models. Experimental results demonstrate that the proposed evaluator achieves a strong correlation (> 0.6) with human judgement and generalizes robustly to diverse responses and corpora. We open-source the code and data in https://github.com/ZHAOTING/dialog-processing.

Automatic Evaluation and Moderation of Open-domain Dialogue Systems

Abstract: The development of Open-Domain Dialogue Systems (ODS)is a trending topic due to the large number of research challenges, large societal and business impact, and advances in the underlying technology. However, the development of these kinds of systems requires two important characteristics:1) automatic evaluation mechanisms that show high correlations with human judgements across multiple dialogue evaluation aspects (with explainable features for providing constructive and explicit feedback on the quality of generative models' responses for quick development and deployment)and 2) mechanisms that can help to control chatbot responses,while avoiding toxicity and employing intelligent ways to handle toxic user comments and keeping interaction flow and engagement. This track at the 10th Dialogue System Technology Challenge (DSTC10) is part of the ongoing effort to promote scalable and toxic-free ODS. This paper describes the datasets and baselines provided to participants, as well as submission evaluation results for each of the two proposed subtasks.

Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References

Abstract: The aim of this paper is to mitigate the shortcomings of automatic evaluation of open-domain dialog systems through multi-reference evaluation. Existing metrics have been shown to correlate poorly with human judgement, particularly in open-domain dialog. One alternative is to collect human annotations for evaluation, which can be expensive and time consuming. To demonstrate the effectiveness of multi-reference evaluation, we augment the test set of DailyDialog with multiple references. A series of experiments show that the use of multiple references results in improved correlation between several automatic metrics and human judgement for both the quality and the diversity of system output.

Human Annotated Dialogues Dataset for Natural Conversational Agents

Abstract: Conversational agents are gaining huge popularity in industrial applications such as digital assistants, chatbots, and particularly systems for natural language understanding (NLU). However, a major drawback is the unavailability of a common metric to evaluate the replies against human judgement for conversational agents. In this paper, we develop a benchmark dataset with human annotations and diverse replies that can be used to develop such metric for conversational agents. The paper introduces a high-quality human annotated movie dialogue dataset, HUMOD, that is developed from the Cornell movie dialogues dataset. This new dataset comprises 28,500 human responses from 9500 multi-turn dialogue history-reply pairs. Human responses include: (i) ratings of the dialogue reply in relevance to the dialogue history; and (ii) unique dialogue replies for each dialogue history from the users. Such unique dialogue replies enable researchers in evaluating their models against six unique human responses for each given history. Detailed analysis on how dialogues are structured and human perception on dialogue score in comparison with existing models are also presented.

An Evaluation Protocol for Generative Conversational Systems

Abstract: There is a multitude of novel generative models for open-domain conversational systems; however, there is no systematic evaluation of different systems. Systematic comparisons require consistency in experimental design, evaluation sets, conversational systems and their outputs, and statistical analysis. We lay out a protocol for the evaluation of conversational models using head-to-head pairwise comparison. We analyze ten recent models that claim state-of-the-art performance using a paired head-to-head performance (win-loss-tie) on five evaluation datasets. Our findings show that DialoGPT and Blender are superior systems using Bradley-Terry model and TrueSkill ranking methods. These findings demonstrate the feasibility of our protocol to evaluate conversational agents and evaluation sets. Finally, we make all code and evaluations publicly available for researchers to compare their model to other state-of-the-art dialog models.

User Response and Sentiment Prediction for Automatic Dialogue Evaluation

Abstract: Automatic evaluation is beneficial for open-domain dialog system development. However, standard word-overlap metrics (BLEU, ROUGE) do not correlate well with human judgements of open-domain dialog systems. In this work we propose to use the sentiment of the next user utterance for turn or dialog level evaluation. Specifically we propose three methods: one that predicts the next sentiment directly, and two others that predict the next user utterance using an utterance or a feedback generator model and then classify its sentiment. Experiments show our model outperforming existing automatic evaluation metrics on both written and spoken open-domain dialogue datasets.

Summary of the PoE evaluation datasets, that are neither involved in training nor tuning. Part of the information are obtained from {{cite:3bdc1a5}} and {{cite:7a9045e}}. * denotes the out-of-domain evaluation datasets w.r.t. the training data. The computation of “Avg.#Utts" includes both the context and the response.

We assess PoE on 16 dialogue evaluation datasets. The selection of evaluation datasets is guided by the recent comprehensive survey on automatic dialogue evaluation metrics {{cite:3bdc1a5}} as well as the "Automatic Evaluation" shared task of DSTC10The Tenth Dialog System Technology Challenge (DSTC10) {{cite:7a9045e}}. Table {{table:5a246ebf-d9c3-42c3-ac18-b8443b95d719}}  summarizes the essential characteristics of all evaluation datasets. Some evaluation datasets contain annotations along multiple evaluation criteria. For example, the FED-Turn {{cite:81b829c}} dataset contains annotations along 9 different fine-grained criteria, such as relevance, interestingness, fluency, etc. Since multi-dimensional evaluation is beyond the scope of this work, we only consider response appropriateness in our analysis. For evaluation datasets without annotations along response appropriateness, the criterion that is closest to response appropriateness is considered, such as context relevance or overall quality. As shown in Table {{table:5a246ebf-d9c3-42c3-ac18-b8443b95d719}} , the “Dimension" column contains the criteria we consider for our correlation analysis.


Building empathetic chatbot for social interaction

Building emotion chatting machine (ECM) for large scale conversation generation

Developing affect-driven dialogue system which generates emotional responses in a controlled manner

Building conversational system emotional by using editing constraints to generate more meaningful and customizable emotional replies

Building open-domain neural dialogue models by augmenting them with affective intelligence.

Building a tone-aware chatbot for customer care on social media.

Building affect-rich open domain human-machine conversation system.

Seq2seq learning with GRU-RNN model which takes into account intelligent quotient (IQ) and emotion quotien (EQ).

Seq2seq learning with GRU which incorporates emotion detection to capture implicit internal emotion states.

Seq2seq with GRU which incorporates emotion classifier, also affective re-ranking in the last step to produce the response.

Encoder-decoder architecture with reinforcement learning approach by using asynchronous decoder which uses keyword predictor to predict the topic and emotion editor to produce emotional-embedded response.

Encoder-decoder architecture with LSTM model using cognitively engineered affective word embedding, and also affectively diverse beam search for decoding

Encoder-decoder architecture with LSTM, which the decoder is modified to handle meta information by add an embedding vector as tone indicator to produce several tone responses including passionate, empathetic, and neutral.

Extends Seq2Seq model and adopts VAD (Valence, Arousal and Dominance) affective notations. Proposed model also considers the effect of negators and intensifiers via a novel affective attention mechanism.

The Design and Implementation of XiaoIce, an Empathetic Social Chatbot

Abstract: This article describes the development of Microsoft XiaoIce, the most popular social chatbot in the world. XiaoIce is uniquely designed as an artifical intelligence companion with an emotional connection to satisfy the human need for communication, affection, and social belonging. We take into account both intelligent quotient and emotional quotient in system design, cast human–machine social chat as decision-making over Markov Decision Processes, and optimize XiaoIce for long-term user engagement, measured in expected Conversation-turns Per Session (CPS). We detail the system architecture and key components, including dialogue manager, core chat, skills, and an empathetic computing module. We show how XiaoIce dynamically recognizes human feelings and states, understands user intent, and responds to user needs throughout long conversations. Since the release in 2014, XiaoIce has communicated with over 660 million active users and succeeded in establishing long-term relationships with many of them. Analysis of large-scale online logs shows that XiaoIce has achieved an average CPS of 23, which is significantly higher than that of other chatbots and even human conversations.

Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory

Abstract: Perception and expression of emotion are key factors to the success of dialogue systems or conversational agents. However, this problem has not been studied in large-scale conversation generation so far. In this paper, we propose Emotional Chatting Machine (ECM) that can generate appropriate responses not only in content (relevant and grammatical) but also in emotion (emotionally consistent). To the best of our knowledge, this is the first work that addresses the emotion factor in large-scale conversation generation. ECM addresses the factor using three new mechanisms that respectively (1) models the high-level abstraction of emotion expressions by embedding emotion categories, (2) captures the change of implicit internal emotion states, and (3) uses explicit emotion expressions with an external emotion vocabulary. Experiments show that the proposed model can generate responses appropriate not only in content but also in emotion.

Affect-Driven Dialog Generation

Abstract: The majority of current systems for end-to-end dialog generation focus on response quality without an explicit control over the affective content of the responses. In this paper, we present an affect-driven dialog system, which generates emotional responses in a controlled manner using a continuous representation of emotions. The system achieves this by modeling emotions at a word and sequence level using: (1) a vector representation of the desired emotion, (2) an affect regularizer, which penalizes neutral words, and (3) an affect sampling method, which forces the neural network to generate diverse words that are emotionally relevant. During inference, we use a re-ranking procedure that aims to extract the most emotionally relevant responses using a human-in-the-loop optimization process. We study the performance of our system in terms of both quantitative (BLEU score and response diversity), and qualitative (emotional appropriateness) measures.

Reinforcement Learning Based Emotional Editing Constraint Conversation Generation

Abstract: In recent years, the generation of conversation content based on deep neural networks has attracted many researchers. However, traditional neural language models tend to generate general replies, lacking logical and emotional factors. This paper proposes a conversation content generation model that combines reinforcement learning with emotional editing constraints to generate more meaningful and customizable emotional replies. The model divides the replies into three clauses based on pre-generated keywords and uses the emotional editor to further optimize the final reply. The model combines multi-task learning with multiple indicator rewards to comprehensively optimize the quality of replies. Experiments shows that our model can not only improve the fluency of the replies, but also significantly enhance the logical relevance and emotional relevance of the replies.

Affective Neural Response Generation

Abstract: Existing neural conversational models process natural language primarily on a lexico-syntactic level, thereby ignoring one of the most crucial components of human-to-human dialogue: its affective content. We take a step in this direction by proposing three novel ways to incorporate affective/emotional aspects into long short term memory (LSTM) encoder-decoder neural conversation models: (1) affective word embeddings, which are cognitively engineered, (2) affect-based objective functions that augment the standard cross-entropy loss, and (3) affectively diverse beam search for decoding. Experiments show that these techniques improve the open-domain conversational prowess of encoder-decoder networks by enabling them to produce emotionally rich responses that are more interesting and natural.

Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media

Abstract: Chatbot has become an important solution to rapidly increasing customer care demands on social media in recent years. However, current work on chatbot for customer care ignores a key to impact user experience - tones. In this work, we create a novel tone-aware chatbot that generates toned responses to user requests on social media. We first conduct a formative research, in which the effects of tones are studied. Significant and various influences of different tones on user experience are uncovered in the study. With the knowledge of effects of tones, we design a deep learning based chatbot that takes tone information into account. We train our system on over 1.5 million real customer care conversations collected from Twitter. The evaluation reveals that our tone-aware chatbot generates as appropriate responses to user requests as human agents. More importantly, our chatbot is perceived to be even more empathetic than human agents.

An Affect-Rich Neural Conversational Model with Biased Attention and Weighted Cross-Entropy Loss

Abstract: Affect conveys important implicit information in human communication. Having the capability to correctly express affect during human-machine conversations is one of the major milestones in artificial intelligence. In recent years, extensive research on open-domain neural conversational models has been conducted. However, embedding affect into such models is still under explored. In this paper, we propose an end-to-end affect-rich open-domain neural conversational model that produces responses not only appropriate in syntax and semantics, but also with rich affect. Our model extends the Seq2Seq model and adopts VAD (Valence, Arousal and Dominance) affective notations to embed each word with affects. In addition, our model considers the effect of negators and intensifiers via a novel affective attention mechanism, which biases attention towards affect-rich words in input sentences. Lastly, we train our model with an affect-incorporated objective function to encourage the generation of affect-rich words in the output responses. Evaluations based on both perplexity and human evaluations show that our model outperforms the state-of-the-art baseline model of comparable size in producing natural and affect-rich responses.

Summarization of the Proposed Approaches for Emotionally-Aware Chatbot Development.

Building Emotionally-Aware Chatbot (EAC)

As we mentioned before that emotion is an essential aspect of building humanize chatbot. The rise of the emotionally-aware chatbot is started by Parry {{cite:fc20436}} in early 1975. Now, most of EAC development exploits neural-based model. In this section, we will try to review previous works which focus on EAC development. Table {{table:59cb24c0-aebb-4cea-8e74-381d981b0f81}}  summarizes this information includes the objective and exploited approach of each work. In early development, EAC is designed by using a rule-based approach. However, in recent years mostly EAC exploit neural-based approach. Studies in EAC development become a hot topic start from 2017, noted by the first shared task in Emotion Generation Challenge on NLPCC 2017 {{cite:e0d94b7}}. Based on Table {{table:59cb24c0-aebb-4cea-8e74-381d981b0f81}}  this research line continues to gain massive attention from scholars in the latest years.

Based on Table {{table:59cb24c0-aebb-4cea-8e74-381d981b0f81}} , we can see that most of all recent EAC was built by using encoder-decoder architecture with sequence-to-sequence learning. These seq2seq learning models maximize the likelihood of response and are prepared to incorporate rich data to generate an appropriate answer. Basic seq2seq architecture structured of two recurrent neural networks (RNNs), one as an encoder processing the input and one as a decoder generating the response. long short term memory (LSTM) or gated recurrent unit (GRU) was the most dominant variant of RNNs which used to learn the conversational dataset in these models. Some studies also tried to model this task as a reinforcement learning task, in order to get more generic responses and let the chatbot able to achieve successful long-term conversation. Attention mechanism was also introduced in this reporthttp://web.stanford.edu/class/cs224s/reports/Honghao_Wei.pdf. This mechanism will allow the decoder to focus only on some important parts in the input at every decoding step.

Another vital part of building EAC is emotion classifier to detect emotion contained in the text to produce a more meaningful response. Emotion detection is a well-established task in natural language processing research area. This task was promoted in two latest series of SemEval-2018 (Task 1) and SemEval-2019 (Task 3). Some tasks were focusing on classifying utterance into several categories of emotion {{cite:cc863a4}}. However, there is also a task which trying to predict the emotion intensities contained in the text {{cite:e95720b}}. In the early development of emotion classifier, most of the studies proposed to use traditional machine-learning approach. However, the neural-based approach is able to gain better performance, which leads more scholars to exploit it to deal with this task. In chatbot, the system will generate several responses based on several emotion categories. Then the system will respond with the most appropriate emotion based on emotion detected on posted utterance by emotion classifier. Based on Table {{table:59cb24c0-aebb-4cea-8e74-381d981b0f81}} , studies have different emotion categories based on their focus and objective in building chatbots.


Empirical methods aiming at known objects

Empirical methods aiming at unknown objects

Utilize the pose and transform grasp points from know objects to the partial data

Regress the grasp points by training CNN networks

Inherit from object detection deep learning networks

Multi-view self-supervised deep learning for 6D pose estimation in the Amazon Picking Challenge

Abstract: Robot warehouse automation has attracted significant interest in recent years, perhaps most visibly in the Amazon Picking Challenge (APC) [1]. A fully autonomous warehouse pick-and-place system requires robust vision that reliably recognizes and locates objects amid cluttered environments, self-occlusions, sensor noise, and a large variety of objects. In this paper we present an approach that leverages multiview RGB-D data and self-supervised, data-driven learning to overcome those difficulties. The approach was part of the MIT-Princeton Team system that took 3rd- and 4th-place in the stowing and picking tasks, respectively at APC 2016. In the proposed approach, we segment and label multiple views of a scene with a fully convolutional neural network, and then fit pre-scanned 3D object models to the resulting segmentation to get the 6D object pose. Training a deep neural network for segmentation typically requires a large amount of training data. We propose a self-supervised method to generate a large labeled dataset without tedious manual segmentation. We demonstrate that our system can reliably estimate the 6D pose of objects under a variety of scenarios. All code, data, and benchmarks are available at http://apc.cs.princeton.edu/

Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Abstract: To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table. We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8sec with a success rate of 93% on eight known objects with adversarial geometry and is 3x faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99% precision (one false positive out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net .

Deep learning for detecting robotic grasps

Abstract: We consider the problem of detecting robotic grasps in an RGB-D view of a scene containing objects. In this work, we apply a deep learning approach to solve this problem, which avoids time-consuming hand-design of features. This presents two main challenges. First, we need to evaluate a huge number of candidate grasps. In order to make detection fast and robust, we present a two-step cascaded system with two deep networks, where the top detections from the first are re-evaluated by the second. The first network has fewer features, is faster to run, and can effectively prune out unlikely candidate grasps. The second, with more features, is slower but has to run only on the top few detections. Second, we need to handle multimodal inputs effectively, for which we present a method that applies structured regularization on the weights based on multimodal group regularization. We show that our method improves performance on an RGBD robotic grasping dataset, and can be used to successfully execute grasps on two different robotic platforms.

Summary of grasp detection methods.

Grasp detection is defined as being able to recognize the grasping points or the grasping pose for an object in any given images {{cite:432443d}}. As stated in {{cite:169805d}}, a grasping strategy should ensure stability, task compatibility and adaptability to novel objects, and the grasp quality can be measured with the location of contact points on the object and the hand configuration {{cite:50e0c0f}}. There exist analytical approaches and empirical approaches in order to grasp a novel object and accomplish a following task. Analytical methods choose the finger positions and the hand configuration with kinematical and dynamical formulations of the grasp stability or the task requirements, and empirical methods use learning algorithms to choose a grasp that depend on the specific task and the target object's geometry. Based on whether or not the object localization, the object pose is required, we further divide them into three kinds, as shown in Table {{table:66000bd4-9c50-46ee-b3ce-3d2ad2dc7c32}} . Methods with known localization and pose will be discussed in this section. Methods with known localization and without pose, methods without localization and without pose, will be discussed in next two sections, respectively.


Binary (ulcer/non-ulcer endoscopy image)

Binary (considered pos/neg cases for different labels like wound, infection, granulation, etc.

WoundNet (modified version of VGG-16) and Deepwound (an ensemble model).

DFU dataset with 397 images (292 abnormal, 105 normal cases).

1875 images obtained from wireless capsule endoscopy video frames.

1335 wound images collected via smartphones and internet.

For small DFUs and DFUs having similar color like surrounding skins is hard to classify by this network. This also goes for normal skins with wrinkle and high red tones.

An unbalanced test set (3:1 ratio) has been used.

As accuracy varies from 72% to 97% for different binary classes, for some specific classes (like: drainage (72%)) this model does not work well.

DFUNet: Convolutional Neural Networks for Diabetic Foot Ulcer Classification

Abstract: Globally, in 2016, one out of eleven adults suffered from Diabetes Mellitus. Diabetic Foot Ulcers (DFU) are a major complication of this disease, which if not managed properly can lead to amputation. Current clinical approaches to DFU treatment rely on patient and clinician vigilance, which has significant limitations such as the high cost involved in the diagnosis, treatment and lengthy care of the DFU. We collected an extensive dataset of foot images, which contain DFU from different patients. In this paper, we have proposed the use of traditional computer vision features for detecting foot ulcers among diabetic patients, which represent a cost-effective, remote and convenient healthcare solution. Furthermore, we used Convolutional Neural Networks (CNNs) for the first time in DFU classification. We have proposed a novel convolutional neural network architecture, DFUNet, with better feature extraction to identify the feature differences between healthy skin and the DFU. Using 10-fold cross-validation, DFUNet achieved an AUC score of 0.962. This outperformed both the machine learning and deep learning classifiers we have tested. Here we present the development of a novel and highly sensitive DFUNet for objectively detecting the presence of DFUs. This novel approach has the potential to deliver a paradigm shift in diabetic foot care.

Application of Convolutional Neural Networks for Automated Ulcer Detection in Wireless Capsule Endoscopy Images

Abstract: Detection of abnormalities in wireless capsule endoscopy (WCE) images is a challenging task. Typically, these images suffer from low contrast, complex background, variations in lesion shape and color, which affect the accuracy of their segmentation and subsequent classification. This research proposes an automated system for detection and classification of ulcers in WCE images, based on state-of-the-art deep learning networks. Deep learning techniques, and in particular, convolutional neural networks (CNNs), have recently become popular in the analysis and recognition of medical images. The medical image datasets used in this study were obtained from WCE video frames. In this work, two milestone CNN architectures, namely the AlexNet and the GoogLeNet are extensively evaluated in object classification into ulcer or non-ulcer. Furthermore, we examine and analyze the images identified as containing ulcer objects to evaluate the efficiency of the utilized CNNs. Extensive experiments show that CNNs deliver superior performance, surpassing traditional machine learning methods by large margins, which supports their effectiveness as automated diagnosis tools.

Deepwound: Automated Postoperative Wound Assessment and Surgical Site Surveillance through Convolutional Neural Networks

Abstract: Postoperative wound complications are a significant cause of expense for hospitals, doctors, and patients. Hence, an effective method to diagnose the onset of wound complications is strongly desired. Algorithmically classifying wound images is a difficult task due to the variability in the appearance of wound sites. Convolutional neural networks (CNNs), a subgroup of artificial neural networks that have shown great promise in analyzing visual imagery, can be leveraged to categorize surgical wounds. We present a multi-label CNN ensemble, Deepwound, trained to classify wound images using only image pixels and corresponding labels as inputs. Our final computational model can accurately identify the presence of nine labels: drainage, fibrinous exudate, granulation tissue, surgical site infection, open wound, staples, steri strips, and sutures. Our model achieves receiver operating curve (ROC) area under curve (AUC) scores, sensitivity, specificity, and F1 scores superior to prior work in this area. Smartphones provide a means to deliver accessible wound care due to their increasing ubiquity. Paired with deep neural networks, they offer the capability to provide clinical insight to assist surgeons during postoperative care. We also present a mobile application frontend to Deepwound that assists patients in tracking their wound and surgical recovery from the comfort of their home.

Summary of Wound Image Classification Works.

Table {{table:093e104c-381a-41a5-8c6b-ed308f77aac3}}  summarizes the reviewed studies. Only a few papers were identified that discuss wound analysis from the wound type classification point of view. Also, most of the publications on wound type classification, discuss only the binary classification problems such as classifying the samples into normal and abnormal cases. Having difficulties to access a reliable dataset can be mentioned as a reason for this issue. Providing data to fill this gap in the literature was one of the motivations for our research. Moreover, many papers discussed only the patch-wise or ROI classification instead of the image-wise wound classification. In the rest of this paper, we propose an ensemble classifier for image-wise multi-class wound type classification using deep convolutional neural networks.


Cars from web-nature and surveillance-nature

Improved version of iNat2017 {{cite:fccc1da}}

Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

A large-scale car dataset for fine-grained categorization and verification

Abstract: This paper aims to highlight vision related tasks centered around “car”, which has been largely neglected by vision community in comparison to other objects. We show that there are still many interesting car-related problems and applications, which are not yet well explored and researched. To facilitate future car-related research, in this paper we present our on-going effort in collecting a large-scale dataset, “CompCars”, that covers not only different car views, but also their different internal and external parts, and rich attributes. Importantly, the dataset is constructed with a cross-modality nature, containing a surveillance-nature set and a web-nature set. We further demonstrate a few important applications exploiting the dataset, namely car model classification, car model verification, and attribute prediction. We also discuss specific challenges of the car-related problems and other potential applications that worth further investigations. The latest dataset can be downloaded at http://mmlab.ie.cuhk.edu.hk/ datasets/comp_cars/index.html.

Benchmarking Representation Learning for Natural World Image Collections

Abstract: Recent progress in self-supervised learning has resulted in models that are capable of extracting rich representations from image collections without requiring any explicit label supervision. However, to date the vast majority of these approaches have restricted themselves to training on standard benchmark datasets such as ImageNet. We argue that fine-grained visual categorization problems, such as plant and animal species classification, provide an informative testbed for self-supervised learning. In order to facilitate progress in this area we present two new natural world visual classification datasets, iNat2021 and NeWT. The former consists of 2.7M images from 10k different species up-loaded by users of the citizen science application iNaturalist. We designed the latter, NeWT, in collaboration with domain experts with the aim of benchmarking the performance of representation learning algorithms on a suite of challenging natural world binary classification tasks that go beyond standard species classification. These two new datasets allow us to explore questions related to large-scale representation and transfer learning in the context of fine-grained categories. We provide a comprehensive analysis of feature extractors trained with and without supervision on ImageNet and iNat2021, shedding light on the strengths and weaknesses of different learned features across a diverse set of tasks. We find that features produced by standard supervised methods still outperform those produced by self-supervised approaches such as SimCLR. However, improved self-supervised learning methods are constantly being released and the iNat2021 and NeWT datasets are a valuable resource for tracking their progress.

Summarization and comparisons of 13 widely-used part segmentation benchmark datasets. The Bbox and Part in annotation items indicate that the dataset provides object bounding box labels and part-level localization labels respectively.

In this subsection, we summarize the prevailing benchmark datasets in the field of fine-grained recognition. In Tab. {{table:28bb2a27-788d-4c86-963a-07f0a91602c3}} , with the development of machine learning systems, earlier works have established benchmark datasets with more than 100 categories for the classification of common daily objects, including Oxford 102 Flowers {{cite:8de82a0}} for plants, CUB-200-2011 {{cite:ffd63b0}} for more than 200 bird categories, and Stanford-Dogs {{cite:866f5d5}} for the classification of 120 dog sub-categories.


Better convergence pattern and scalability to generate high-res image

Improve distribution modeling and sample quality

Capture diverse, complementary and interpretable representations

Extends vanilla GAN, Multi-modal mapping

Generalize cGAN, unsupervised disentangled representation learning

Extends traditional GANs to Bayesian framework, naturally handles unsupervised, supervised and semi-supervised learning problems

Removes vanishing gradients, partially removes mode collapse

Effective in modeling long-range stability

Reduce generation time, improve generation quality

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

Abstract: In this paper we introduce a generative parametric model capable of producing high quality samples of natural images. Our approach uses a cascade of convolutional networks within a Laplacian pyramid framework to generate images in a coarse-to-fine fashion. At each level of the pyramid, a separate generative convnet model is trained using the Generative Adversarial Nets (GAN) approach (Goodfellow et al.). Samples drawn from our model are of significantly higher quality than alternate approaches. In a quantitative assessment by human evaluators, our CIFAR10 samples were mistaken for real images around 40% of the time, compared to 10% for samples drawn from a GAN baseline model. We also show samples from models trained on the higher resolution images of the LSUN scene dataset.

Abstract: We introduce the "Energy-based Generative Adversarial Network" model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions. Similar to the probabilistic GANs, a generator is seen as being trained to produce contrastive samples with minimal energies, while the discriminator is trained to assign high energies to these generated samples. Viewing the discriminator as an energy function allows to use a wide variety of architectures and loss functionals in addition to the usual binary classifier with logistic output. Among them, we show one instantiation of EBGAN framework as using an auto-encoder architecture, with the energy being the reconstruction error, in place of the discriminator. We show that this form of EBGAN exhibits more stable behavior than regular GANs during training. We also show that a single-scale architecture can be trained to generate high-resolution images.

BEGAN: Boundary Equilibrium Generative Adversarial Networks

Abstract: We propose a new equilibrium enforcing method paired with a loss derived from the Wasserstein distance for training auto-encoder based Generative Adversarial Networks. This method balances the generator and discriminator during training. Additionally, it provides a new approximate convergence measure, fast and stable training and high visual quality. We also derive a way of controlling the trade-off between image diversity and visual quality. We focus on the image generation task, setting a new milestone in visual quality, even at higher resolutions. This is achieved while using a relatively simple model architecture and a standard training procedure.

Bayesian GAN

Abstract: Generative adversarial networks (GANs) can implicitly learn rich distributions over images, audio, and data which are hard to model with an explicit likelihood. We present a practical Bayesian formulation for unsupervised and semi-supervised learning with GANs. Within this framework, we use stochastic gradient Hamiltonian Monte Carlo to marginalize the weights of the generator and discriminator networks. The resulting approach is straightforward and obtains good performance without any standard interventions such as feature matching, or mini-batch discrimination. By exploring an expressive posterior over the parameters of the generator, the Bayesian GAN avoids mode-collapse, produces interpretable and diverse candidate samples, and provides state-of-the-art quantitative results for semi-supervised learning on benchmarks including SVHN, CelebA, and CIFAR-10, outperforming DCGAN, Wasserstein GANs, and DCGAN ensembles.

f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization

Abstract: Generative neural samplers are probabilistic models that implement sampling using feedforward neural networks: they take a random input vector and produce a sample from a probability distribution defined by the network weights. These models are expressive and allow efficient computation of samples and derivatives, but cannot be used for computing likelihoods or for marginalization. The generative-adversarial training method allows to train such models through the use of an auxiliary discriminative neural network. We show that the generative-adversarial approach is a special case of an existing more general variational divergence estimation approach. We show that any f-divergence can be used for training generative neural samplers. We discuss the benefits of various choices of divergence functions on training complexity and the quality of the obtained generative models.

Conditional Generative Adversarial Nets

Abstract: Generative Adversarial Nets [8] were recently introduced as a novel way to train generative models. In this work we introduce the conditional version of generative adversarial nets, which can be constructed by simply feeding the data, y, we wish to condition on to both the generator and discriminator. We show that this model can generate MNIST digits conditioned on class labels. We also illustrate how this model could be used to learn a multi-modal model, and provide preliminary examples of an application to image tagging in which we demonstrate how this approach can generate descriptive tags which are not part of training labels.

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Abstract: This paper describes InfoGAN, an information-theoretic extension to the Generative Adversarial Network that is able to learn disentangled representations in a completely unsupervised manner. InfoGAN is a generative adversarial network that also maximizes the mutual information between a small subset of the latent variables and the observation. We derive a lower bound to the mutual information objective that can be optimized efficiently, and show that our training procedure can be interpreted as a variation of the Wake-Sleep algorithm. Specifically, InfoGAN successfully disentangles writing styles from digit shapes on the MNIST dataset, pose from lighting of 3D rendered images, and background digits from the central digit on the SVHN dataset. It also discovers visual concepts that include hair styles, presence/absence of eyeglasses, and emotions on the CelebA face dataset. Experiments show that InfoGAN learns interpretable representations that are competitive with representations learned by existing fully supervised methods.

Bayesian Conditional Generative Adverserial Networks

Abstract: Traditional GANs use a deterministic generator function (typically a neural network) to transform a random noise input $z$ to a sample $\mathbf{x}$ that the discriminator seeks to distinguish. We propose a new GAN called Bayesian Conditional Generative Adversarial Networks (BC-GANs) that use a random generator function to transform a deterministic input $y'$ to a sample $\mathbf{x}$. Our BC-GANs extend traditional GANs to a Bayesian framework, and naturally handle unsupervised learning, supervised learning, and semi-supervised learning problems. Experiments show that the proposed BC-GANs outperforms the state-of-the-arts.

Abstract: Unsupervised learning with generative adversarial networks (GANs) has proven hugely successful. Regular GANs hypothesize the discriminator as a classifier with the sigmoid cross entropy loss function. However, we found that this loss function may lead to the vanishing gradients problem during the learning process. To overcome such a problem, we propose in this paper the Least Squares Generative Adversarial Networks (LSGANs) which adopt the least squares loss function for the discriminator. We show that minimizing the objective function of LSGAN yields minimizing the Pearson $\chi^2$ divergence. There are two benefits of LSGANs over regular GANs. First, LSGANs are able to generate higher quality images than regular GANs. Second, LSGANs perform more stable during the learning process. We evaluate LSGANs on five scene datasets and the experimental results show that the images generated by LSGANs are of better quality than the ones generated by regular GANs. We also conduct two comparison experiments between LSGANs and regular GANs to illustrate the stability of LSGANs.

Wasserstein GAN

Abstract: We introduce a new algorithm named WGAN, an alternative to traditional GAN training. In this new model, we show that we can improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches. Furthermore, we show that the corresponding optimization problem is sound, and provide extensive theoretical work highlighting the deep connections to other distances between distributions.

Improved Training of Wasserstein GANs

Abstract: Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.

McGan: Mean and Covariance Feature Matching GAN

Abstract: We introduce new families of Integral Probability Metrics (IPM) for training Generative Adversarial Networks (GAN). Our IPMs are based on matching statistics of distributions embedded in a finite dimensional feature space. Mean and covariance feature matching IPMs allow for stable training of GANs, which we will call McGan. McGan minimizes a meaningful loss between distributions.

The Cramer Distance as a Solution to Biased Wasserstein Gradients

Abstract: The Wasserstein probability metric has received much attention from the machine learning community. Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes. The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling. In this paper we describe three natural properties of probability divergences that reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients. The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third. We provide empirical evidence suggesting that this is a serious issue in practice. Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the Cram\'er distance. We show that the Cram\'er distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences. To illustrate the relevance of the Cram\'er distance in practice we design a new algorithm, the Cram\'er Generative Adversarial Network (GAN), and show that it performs significantly better than the related Wasserstein GAN.

Fisher GAN

Abstract: Generative Adversarial Networks (GANs) are powerful models for learning complex distributions. Stable training of GANs has been addressed in many recent works which explore different metrics between distributions. In this paper we introduce Fisher GAN which fits within the Integral Probability Metrics (IPM) framework for training GANs. Fisher GAN defines a critic with a data dependent constraint on its second order moments. We show in this paper that Fisher GAN allows for stable and time efficient training that does not compromise the capacity of the critic, and does not need data independent constraints such as weight clipping. We analyze our Fisher IPM theoretically and provide an algorithm based on Augmented Lagrangian for Fisher GAN. We validate our claims on both image sample generation and semi-supervised classification using Fisher GAN.

MAGAN: Margin Adaptation for Generative Adversarial Networks

Abstract: We propose the Margin Adaptation for Generative Adversarial Networks (MAGANs) algorithm, a novel training procedure for GANs to improve stability and performance by using an adaptive hinge loss function. We estimate the appropriate hinge loss margin with the expected energy of the target distribution, and derive principled criteria for when to update the margin. We prove that our method converges to its global optimum under certain assumptions. Evaluated on the task of unsupervised image generation, the proposed training procedure is simple yet robust on a diverse set of data, and achieves qualitative and quantitative improvements compared to the state-of-the-art.

Self-Attention Generative Adversarial Networks

Abstract: In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other. Furthermore, recent work has shown that generator conditioning affects GAN performance. Leveraging this insight, we apply spectral normalization to the GAN generator and find that this improves training dynamics. The proposed SAGAN achieves the state-of-the-art results, boosting the best published Inception score from 36.8 to 52.52 and reducing Frechet Inception distance from 27.62 to 18.65 on the challenging ImageNet dataset. Visualization of the attention layers shows that the generator leverages neighborhoods that correspond to object shapes rather than local regions of fixed shape.

Improving the Speed and Quality of GAN by Adversarial Training

Abstract: Generative adversarial networks (GAN) have shown remarkable results in image generation tasks. High fidelity class-conditional GAN methods often rely on stabilization techniques by constraining the global Lipschitz continuity. Such regularization leads to less expressive models and slower convergence speed; other techniques, such as the large batch training, require unconventional computing power and are not widely accessible. In this paper, we develop an efficient algorithm, namely FastGAN (Free AdverSarial Training), to improve the speed and quality of GAN training based on the adversarial training technique. We benchmark our method on CIFAR10, a subset of ImageNet, and the full ImageNet datasets. We choose strong baselines such as SNGAN and SAGAN; the results demonstrate that our training algorithm can achieve better generation quality (in terms of the Inception score and Frechet Inception distance) with less overall training time. Most notably, our training algorithm brings ImageNet training to the broader public by requiring 2-4 GPUs.

Major GAN variants.

The original GAN formulation cannot be directly used to solve challenging problems in computer vision and other fields. There have been several advancements in how we use GANs, either modifications to the generator network and/or the discriminator network, its formulation or tweaks to the loss functions to speed up the GAN training process and/or to obtain stable results. A few of the significant variants are listed in the table {{table:ad8a2e19-d862-4cbf-bee1-869a89d81586}} .


Bayesian U-Net. Episdemic uncertainty estimations and post processing

Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery

Abstract: Obtaining models that capture imaging markers relevant for disease progression and treatment monitoring is challenging. Models are typically based on large amounts of data with annotated examples of known markers aiming at automating detection. High annotation effort and the limitation to a vocabulary of known markers limit the power of such approaches. Here, we perform unsupervised learning to identify anomalies in imaging data as candidates for markers. We propose AnoGAN, a deep convolutional generative adversarial network to learn a manifold of normal anatomical variability, accompanying a novel anomaly scoring scheme based on the mapping from image space to a latent space. Applied to new data, the model labels anomalies, and scores image patches indicating their fit into the learned distribution. Results on optical coherence tomography images of the retina demonstrate that the approach correctly identifies anomalous images, such as images containing retinal fluid or hyperreflective foci.

Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCT

Abstract: Diagnosis and treatment guidance are aided by detecting relevant biomarkers in medical images. Although supervised deep learning can perform accurate segmentation of pathological areas, it is limited by requiring a priori definitions of these regions, large-scale annotations, and a representative patient cohort in the training set. In contrast, anomaly detection is not limited to specific definitions of pathologies and allows for training on healthy samples without annotation. Anomalous regions can then serve as candidates for biomarker discovery. Knowledge about normal anatomical structure brings implicit information for detecting anomalies. We propose to take advantage of this property using Bayesian deep learning, based on the assumption that epistemic uncertainties will correlate with anatomical deviations from a normal training set. A Bayesian U-Net is trained on a well-defined healthy environment using weak labels of healthy anatomy produced by existing methods. At test time, we capture epistemic uncertainty estimates of our model using Monte Carlo dropout. A novel post-processing technique is then applied to exploit these estimates and transfer their layered appearance to smooth blob-shaped segmentations of the anomalies. We experimentally validated this approach in retinal optical coherence tomography (OCT) images, using weak labels of retinal layers. Our method achieved a Dice index of 0.789 in an independent anomaly test set of age-related macular degeneration (AMD) cases. The resulting segmentations allowed very high accuracy for separating healthy and diseased cases with late wet AMD, dry geographic atrophy (GA), diabetic macular edema (DME) and retinal vein occlusion (RVO). Finally, we qualitatively observed that our approach can also detect other deviations in normal scans such as cut edge artifacts.

Table consisting of ocular image based results obtained by the literature review

The semi-exhaustive literature review resulted in 45 research items, from which further 6 were removed due to not containing applications in medical imaging (only exemplar stated in abstract) or being non-available. The resulting papers were further clustered into 5 categories (corresponding to Tab. {{table:586d8cd2-d882-4536-a90e-d9e6b792e45b}} -{{table:6b585252-d90d-46db-94b0-1b1db14ac7f6}}  by their imaging methods. Tab. {{table:586d8cd2-d882-4536-a90e-d9e6b792e45b}}  shows papers applying anomaly detection to occular medical images with retinal fundus images and optical coherence tomography (OCT). Tab. {{table:2f8ce732-b2a8-4848-afc9-5dff2b3298fa}}  focuses on papers with applications in the center body region, with chest X-rays and mammography. Tab. {{table:758a0f69-7649-482d-bc91-f5a73ad646b1}}  summarizes application papers, using CT and functional magnetic resonance imaging (fMRI). Tab. {{table:fac491b7-86b6-4824-beb9-d4b963abc220}}  displays papers applying ML to brain Magnetic resonance imaging (MRI) datasets. Tab. {{table:6b585252-d90d-46db-94b0-1b1db14ac7f6}}  shows mixed applications from the domains of breast ultrasound, chest radiographs, histology and fundus images as well as multi-spectral imaging (MSI).


Diff-TTS: A Denoising Diffusion Model for Text-to-Speech

Abstract: Although neural text-to-speech (TTS) models have attracted a lot of attention and succeeded in generating human-like speech, there is still room for improvements to its naturalness and architectural efficiency. In this work, we propose a novel non-autoregressive TTS model, namely Diff-TTS, which achieves highly natural and efficient speech synthesis. Given the text, Diff-TTS exploits a denoising diffusion framework to transform the noise signal into a mel-spectrogram via diffusion time steps. In order to learn the mel-spectrogram distribution conditioned on the text, we present a likelihood-based optimization method for TTS. Furthermore, to boost up the inference speed, we leverage the accelerated sampling method that allows Diff-TTS to generate raw waveforms much faster without significantly degrading perceptual quality. Through experiments, we verified that Diff-TTS generates 28 times faster than the real-time with a single NVIDIA 2080Ti GPU.

Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech

Abstract: Recently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by gradually transforming noise predicted by encoder and aligned with text input by means of Monotonic Alignment Search. The framework of stochastic differential equations helps us to generalize conventional diffusion probabilistic models to the case of reconstructing data from noise with different parameters and allows to make this reconstruction flexible by explicitly controlling trade-off between sound quality and inference speed. Subjective human evaluation shows that Grad-TTS is competitive with state-of-the-art text-to-speech approaches in terms of Mean Opinion Score. We will make the code publicly available shortly.

ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech

Abstract: Denoising diffusion probabilistic models (DDPMs) have recently achieved leading performances in many generative tasks. However, the inherited iterative sampling process costs hinder their applications to text-to-speech deployment. Through the preliminary study on diffusion model parameterization, we find that previous gradient-based TTS models require hundreds or thousands of iterations to guarantee high sample quality, which poses a challenge for accelerating sampling. In this work, we propose ProDiff, on progressive fast diffusion model for high-quality text-to-speech. Unlike previous work estimating the gradient for data density, ProDiff parameterizes the denoising model by directly predicting clean data to avoid distinct quality degradation in accelerating sampling. To tackle the model convergence challenge with decreased diffusion iterations, ProDiff reduces the data variance in the target site via knowledge distillation. Specifically, the denoising model uses the generated mel-spectrogram from an N-step DDIM teacher as the training target and distills the behavior into a new model with N/2 steps. As such, it allows the TTS model to make sharp predictions and further reduces the sampling time by orders of magnitude. Our evaluation demonstrates that ProDiff needs only 2 iterations to synthesize high-fidelity mel-spectrograms, while it maintains sample quality and diversity competitive with state-of-the-art models using hundreds of steps. ProDiff enables a sampling speed of 24x faster than real-time on a single NVIDIA 2080Ti GPU, making diffusion models practically applicable to text-to-speech synthesis deployment for the first time. Our extensive ablation studies demonstrate that each design in ProDiff is effective, and we further show that ProDiff can be easily extended to the multi-speaker setting. Audio samples are available at \url{https://ProDiff.github.io/.}

NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS

Abstract: Expressive text-to-speech (TTS) can synthesize a new speaking style by imiating prosody and timbre from a reference audio, which faces the following challenges: (1) The highly dynamic prosody information in the reference audio is difficult to extract, especially, when the reference audio contains background noise. (2) The TTS systems should have good generalization for unseen speaking styles. In this paper, we present a \textbf{no}ise-\textbf{r}obust \textbf{e}xpressive TTS model (NoreSpeech), which can robustly transfer speaking style in a noisy reference utterance to synthesized speech. Specifically, our NoreSpeech includes several components: (1) a novel DiffStyle module, which leverages powerful probabilistic denoising diffusion models to learn noise-agnostic speaking style features from a teacher model by knowledge distillation; (2) a VQ-VAE block, which maps the style features into a controllable quantized latent space for improving the generalization of style transfer; and (3) a straight-forward but effective parameter-free text-style alignment module, which enables NoreSpeech to transfer style to a textual input from a length-mismatched reference utterance. Experiments demonstrate that NoreSpeech is more effective than previous expressive TTS models in noise environments. Audio samples and code are available at: \href{http://dongchaoyang.top/NoreSpeech\_demo/}{http://dongchaoyang.top/NoreSpeech\_demo/}

Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models

Abstract: We present a novel way of conditioning a pretrained denoising diffusion speech model to produce speech in the voice of a novel person unseen during training. The method requires a short (~3 seconds) sample from the target person, and generation is steered at inference time, without any training steps. At the heart of the method lies a sampling process that combines the estimation of the denoising model with a low-pass version of the new speaker's sample. The objective and subjective evaluations show that our sampling method can generate a voice similar to that of the target speaker in terms of frequency, with an accuracy comparable to state-of-the-art methods, and without training.

Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models

Abstract: There has been a significant progress in Text-To-Speech (TTS) synthesis technology in recent years, thanks to the advancement in neural generative modeling. However, existing methods on any-speaker adaptive TTS have achieved unsatisfactory performance, due to their suboptimal accuracy in mimicking the target speakers’ styles. In this work, we present Grad-StyleSpeech, which is an any-speaker adaptive TTS framework that is based on a diffusion model that can generate highly natural speech with extremely high similarity to target speakers’ voice, given a few seconds of reference speech. Grad-StyleSpeech significantly outperforms recent speaker-adaptive TTS baselines on English benchmarks. Audio samples are available at https://nardien.github.io/grad-stylespeech-demo.

Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data

Abstract: We propose Guided-TTS 2, a diffusion-based generative model for high-quality adaptive TTS using untranscribed data. Guided-TTS 2 combines a speaker-conditional diffusion model with a speaker-dependent phoneme classifier for adaptive text-to-speech. We train the speaker-conditional diffusion model on large-scale untranscribed datasets for a classifier-free guidance method and further fine-tune the diffusion model on the reference speech of the target speaker for adaptation, which only takes 40 seconds. We demonstrate that Guided-TTS 2 shows comparable performance to high-quality single-speaker TTS baselines in terms of speech quality and speaker similarity with only a ten-second untranscribed data. We further show that Guided-TTS 2 outperforms adaptive TTS baselines on multi-speaker datasets even with a zero-shot adaptation setting. Guided-TTS 2 can adapt to a wide range of voices only using untranscribed speech, which enables adaptive TTS with the voice of non-human characters such as Gollum in \textit{"The Lord of the Rings"}.

Experimental results of acoustic model based on diffusion model

Acoustic model that transforms a text to acoustic features is a core component in the the task of text to speech. A summary of representative work that apply diffusion model to acoustic model is shown in Table {{table:6f4a9aa1-4f70-487f-a7f9-638bd816e5b4}} .


End-to-End Learning of Visual Representations From Uncurated Instructional Videos

Abstract: Annotating videos is cumbersome, expensive and not scalable. Yet, many strong video models still rely on manually annotated data. With the recent introduction of the HowTo100M dataset, narrated videos now offer the possibility of learning video representations without manual supervision. In this work we propose a new learning approach, MIL-NCE, capable of addressing mis- alignments inherent in narrated videos. With this approach we are able to learn strong video representations from scratch, without the need for any manual annotation. We evaluate our representations on a wide range of four downstream tasks over eight datasets: action recognition (HMDB-51, UCF-101, Kinetics-700), text-to- video retrieval (YouCook2, MSR-VTT), action localization (YouTube-8M Segments, CrossTask) and action segmentation (COIN). Our method outperforms all published self-supervised approaches for these tasks as well as several fully supervised baselines.

VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

Abstract: We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/examples/MMPT.

UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

Abstract: With the recent success of the pre-training technique for NLP and image-linguistic tasks, some video-linguistic pre-training works are gradually developed to improve video-text related downstream tasks. However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks. This paper proposes UniVL: a Unified Video and Language pre-training model for both multimodal understanding and generation. It comprises four components, including two single-modal encoders, a cross encoder, and a decoder with the Transformer backbone. Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The pre-train is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that the UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks.

COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

Abstract: Many real-world video-text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention-aware feature aggregation layer, which leverages the local temporal context (intra-level, e.g., within a clip), a contextual transformer to learn the interactions between low-level and high-level semantics (inter-level, e.g. clip-video, sentence-paragraph), and a cross-modal cycle-consistency loss to connect video and text. The resulting method compares favorably to the state of the art on several benchmarks while having few parameters. All code is available open-source at this https URL

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

Abstract: Our objective in this work is video-text retrieval – in particular a joint embedding that enables efficient text-to-video retrieval. The challenges in this area include the design of the visual architecture and the nature of the training data, in that the available large scale video-text training datasets, such as HowTo100M, are noisy and hence competitive performance is achieved only at scale through large amounts of compute.We address both these challenges in this paper. We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is an adaptation and extension of the recent ViT and Timesformer architectures, and consists of attention in both space and time. The model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. It is trained with a curriculum learning schedule that begins by treating images as ‘frozen’ snapshots of video, and then gradually learns to attend to increasing temporal context when trained on video datasets. We also provide a new video-text pretraining dataset WebVid-2M, comprised of over two million videos with weak captions scraped from the internet. Despite training on datasets that are an order of magnitude smaller, we show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.

Details of self-supervised video-language models used in this study.

We evaluate the most popular video-language approach MIL-NCE {{cite:2c7ff8a}} which uses a CNN backbone and Word2Vec word embeddings with an MIL-NCE contrastive loss between text-video pairs. We further evaluate models and approaches that utilize visual features from {{cite:2c7ff8a}} with further training and different self-supervised approaches. The more recent method VideoClip {{cite:738a754}} is a transformer-based approach relying instead on BERT {{cite:eaec3b5}} for both text and video encodings with a similar but improved contrastive loss. COOT {{cite:48d945b}} similarly uses transformer-based encoders taking BERT text features and S3D visual features as input and includes cross-attention between the text and video features. Rather than a contrastive loss with negative pairing, COOT focuses on alignment between text and video alone. UniVL {{cite:fbf6cf3}}, is another transformer-based approach that uses a cross-encoding transformer in addition to separate encoders as their self-supervised objective. The final approach evaluated, FIT {{cite:a9b33cf}}, combines image-based research with video. It uses only a small set of frames for a given clip which is encoded using a Visual Transformer (ViT) {{cite:7c63cf7}}, {{cite:f479f8e}}. They also pre-train with a different dataset that comprises of both images from CC3M {{cite:3b9170e}} and video from their own proposed dataset, Web2Vid {{cite:a9b33cf}}. FIT uses a contrastive loss for video-text pairs and for text-video pairs with temporal curriculum learning. More details on these approaches are shown in Table {{table:64f1bb8b-b1f0-4a1c-9737-8eb65e1f2853}} .

To ensure fairness to the original models, we use the official model implementations that were available with pre-trained weights with the same experimental setup as described in these works. These protocols vary between models and datasets. HowTo100M-MIL {{cite:2c7ff8a}} take video as input and split the temporal boundary of the passed video into a clip of 4 with 32 frames for YouCook2 and 16 frames for MSRVTT. They take text as input and embed each word using Word2Vec. VideoClip {{cite:738a754}} and COOT {{cite:48d945b}} use pre-extracted features from the pre-trained S3D-G {{cite:5394732}} model provided by {{cite:2c7ff8a}} while UniVL {{cite:fbf6cf3}} uses pre-extracted features from the same model but before the final layer resulting in a smaller embedding size. VideoClip and UniVL take text as raw input while COOT {{cite:48d945b}} uses pre-extracted text features from BERT {{cite:eaec3b5}}. FIT {{cite:a9b33cf}} splits a clip into 4 segments and randomly selects 1 frame from each. These details are summarized in Table {{table:64f1bb8b-b1f0-4a1c-9737-8eb65e1f2853}} .
We also analyze some models on whether they are fine-tuned, pre-trained or trained from scratch based on the availability of code. In the original implementations, VideoClip, Howto100-MIL and UniVL are pre-trained on HowTo100M {{cite:84d0868}}, COOT was trained from scratch on MSRVTT, and FIT is pre-trained on CC3M {{cite:3b9170e}} and Web2Vid {{cite:a9b33cf}}.
Evaluating models using only pre-trained weights are considered zero-shot (ZS). FIT, VideoClip and UniVL were additionally fine-tuned (FT). Models that are trained on the evaluation datasets without pre-training are considered scratch.


HateBERT: Retraining BERT for Abusive Language Detection in English

Abstract: In this paper, we introduce HateBERT, a re-trained BERT model for abusive language detection in English. The model was trained on RAL-E, a large-scale dataset of Reddit comments in English from communities banned for being offensive, abusive, or hateful that we have collected and made available to the public. We present the results of a detailed comparison between a general pre-trained language model and the abuse-inclined version obtained by retraining with posts from the banned communities on three English datasets for offensive, abusive language and hate speech detection tasks. In all datasets, HateBERT outperforms the corresponding general BERT model. We also discuss a battery of experiments comparing the portability of the generic pre-trained language model and its corresponding abusive language-inclined counterpart across the datasets, indicating that portability is affected by compatibility of the annotated phenomena.

TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification

Abstract: The experimental landscape in natural language processing for social media is too fragmented. Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction. Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domain-specific data. In this paper, we propose a new evaluation framework (TweetEval) consisting of seven heterogeneous Twitter-specific classification tasks. We also provide a strong set of baselines as starting point, and compare different language modeling pre-training strategies. Our initial experiments show the effectiveness of starting off with existing pre-trained generic language models, and continue training them on Twitter corpora.

BERTweet: A pre-trained language model for English Tweets

Abstract: We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having the same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al., 2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification. We release BERTweet under the MIT License to facilitate future research and applications on Tweet data. Our BERTweet is available at https://github.com/VinAIResearch/BERTweet

COVID-Twitter-BERT: A natural language processing model to analyse COVID-19 content on Twitter

Abstract: Introduction This study presents COVID-Twitter-BERT (CT-BERT), a transformer-based model that is pre-trained on a large corpus of COVID-19 related Twitter messages. CT-BERT is specifically designed to be used on COVID-19 content, particularly from social media, and can be utilized for various natural language processing tasks such as classification, question-answering, and chatbots. This paper aims to evaluate the performance of CT-BERT on different classification datasets and compare it with BERT-LARGE, its base model. Methods The study utilizes CT-BERT, which is pre-trained on a large corpus of COVID-19 related Twitter messages. The authors evaluated the performance of CT-BERT on five different classification datasets, including one in the target domain. The model's performance is compared to its base model, BERT-LARGE, to measure the marginal improvement. The authors also provide detailed information on the training process and the technical specifications of the model. Results The results indicate that CT-BERT outperforms BERT-LARGE with a marginal improvement of 10-30% on all five classification datasets. The largest improvements are observed in the target domain. The authors provide detailed performance metrics and discuss the significance of these results. Discussion The study demonstrates the potential of pre-trained transformer models, such as CT-BERT, for COVID-19 related natural language processing tasks. The results indicate that CT-BERT can improve the classification performance on COVID-19 related content, especially on social media. These findings have important implications for various applications, such as monitoring public sentiment and developing chatbots to provide COVID-19 related information. The study also highlights the importance of using domain-specific pre-trained models for specific natural language processing tasks. Overall, this work provides a valuable contribution to the development of COVID-19 related NLP models.

Summary of social-media based T-PTLMs.

T-PTLMs like BERT and RoBERTa are pretrained on formal text. As social media text is highly informal in nature with a lot of noise in the form of irregular grammar, slang words, and non-standard abbreviations, these models have limited performance on social media datasets {{cite:61ed79a}}, {{cite:d60971c}}, {{cite:5563860}}, {{cite:3e6e64d}} . Researchers working at the intersection of social media and NLP have developed social media-specific T-PTLMs either by training from scratch {{cite:d60971c}} or continual pretraining {{cite:61ed79a}}, {{cite:5563860}}, {{cite:3e6e64d}}, {{cite:a1bfa88}}, {{cite:e9d21ed}}, {{cite:840c59f}} and a summary of these models is presented in Table {{table:27db5e32-1529-4f41-9ba2-035749a89968}} . Except for Bertweet {{cite:d60971c}}, all other social media-based T-PTLMs are developed by continual pretraining. Training from scratch is effective only when the pretraining corpus consists of a large number of tweets. Otherwise, continual pretraining is recommended. For example, BERTweet {{cite:d60971c}} is pretrained from scratch using 850M tweets. Barbieri et al. {{cite:3e6e64d}} showed that RoBERTa model trained from scratch on tweets achieved less performance compared to RoBERTa model adapted to social media by continual pretraining. This is because of using just 60M tweets for pretraining. Different from other social media-based T-PTLMs which are developed using commonly used pretraining tasks like MLM and NSP, BERT-SentiX {{cite:840c59f}} is obtained by continual pretraining on user reviews using four novel sentiment aware pretraining tasks.


Mapping the Gnutella Network: Properties of Large-Scale Peer-to-Peer Systems and Implications for System Design

Abstract: Despite recent excitement generated by the peer-to-peer (P2P) paradigm and the surprisingly rapid deployment of some P2P applications, there are few quantitative evaluations of P2P systems behavior. The open architecture, achieved scale, and self-organizing structure of the Gnutella network make it an interesting P2P architecture to study. Like most other P2P applications, Gnutella builds, at the application level, a virtual network with its own routing mechanisms. The topology of this virtual network and the routing mechanisms used have a significant influence on application properties such as performance, reliability, and scalability. We have built a "crawler" to extract the topology of Gnutella's application level network. In this paper we analyze the topology graph and evaluate generated network traffic. Our two major findings are that: (1) although Gnutella is not a pure power-law network, its current configuration has the benefits and drawbacks of a power-law structure, and (2) the Gnutella virtual network topology does not match well the underlying Internet topology, hence leading to ineffective use of the physical networking infrastructure. These findings guide us to propose changes to the Gnutella protocol and implementations that may bring significant performance and scalability improvements. We believe that our findings as well as our measurement and analysis techniques have broad applicability to P2P systems and provide unique insights into P2P system design tradeoffs.

Predicting User Roles in Social Networks using Transfer Learning with Feature Transformation

Abstract: How can we recognise social roles of people, given a completely unlabelled social network? We present a transfer learning approach to network role classification based on feature transformations from each network's local feature distribution to a global feature space. Experiments are carried out on real-world datasets. (See manuscript for the full abstract.)

General description of the five empirical network datasets. The fields in the table are the dataset name, whether the network is directed, the number of nodes (N), the number of edges (m), the meaning of the edges, and the original reference. The name corresponds to the Internal name field in the KONECT database. To access the site to download the dataset, append the internal name to the link http://konect.cc/networks/.

We selected five datasets, all of them extracted from the KONECT network database {{cite:e38312a}}http://konect.cc/, a public online database of more than a one thousand network datasets. The criteria for selecting the networks were that, besides being similar in size to our synthetic networks(10000 nodes), each network had a distinct meaning, i.e., networks representing distinct systems from different contexts. General descriptors of these datasets are displayed on Table {{table:934d9289-b840-4ac0-bb7a-4cf591a2b267}} . There, we can see that we have selected: a social network of friendships among students created from a survey {{cite:48651ff}}, a co-authorship network from the astrophysics section of arXiv from 1995 to 1999 {{cite:cef03ea}}, a citation network of publications from DBLP {{cite:4b642ba}}, a network of connected Gnutella hosts from 2002 {{cite:ccc085c}}, and the communication network of messages among users from the Galician Wikipedia {{cite:0e2a278}}. See  to review the experiments of assumption verification on these empirical datasets.


EEG, ECG, EHRs, PPG, EMG, Speech, NAF, MNIST, Synthetic sets

Nottingham dataset, Midi music files, MIR-1K, TheSession, Speech

Fixed length time series 'vehicle and engine speed'

TSTR, MMD, Reconstruction error, DTW, PCC, IS, FID, ED, S-WD, RMSE, MAE, FD, PRD, Averaging Samples, WA, UAR, MV-DTW

Human perception, Polyphony, Scale Consistency, Tone Span, Repetitions, NSDR, SIR, SAR, FD, t-SNE, Distribution of notes

Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs

Abstract: Generative Adversarial Networks (GANs) have shown remarkable success as a framework for training models to produce realistic-looking data. In this work, we propose a Recurrent GAN (RGAN) and Recurrent Conditional GAN (RCGAN) to produce realistic real-valued multi-dimensional time series, with an emphasis on their application to medical data. RGANs make use of recurrent neural networks (RNNs) in the generator and the discriminator. In the case of RCGANs, both of these RNNs are conditioned on auxiliary information. We demonstrate our models in a set of toy datasets, where we show visually and quantitatively (using sample likelihood and maximum mean discrepancy) that they can successfully generate realistic time-series. We also describe novel evaluation methods for GANs, where we generate a synthetic labelled training dataset, and evaluate on a real test set the performance of a model trained on the synthetic data, and vice-versa. We illustrate with these metrics that RCGANs can generate time-series data useful for supervised training, with only minor degradation in performance on real test data. This is demonstrated on digit classification from ‘serialised’ MNIST and by training an early warning system on a medical dataset of 17,000 patients from an intensive care unit. We further discuss and analyse the privacy concerns that may arise when using RCGANs to generate realistic synthetic medical time series data, and demonstrate results from differentially private training of the RCGAN.

C-RNN-GAN: Continuous recurrent neural networks with adversarial training

Abstract: Generative adversarial networks have been proposed as a way of efficiently training deep generative neural networks. We propose a generative adversarial model that works on continuous sequential data, and apply it by training it on a collection of classical music. We conclude that it generates music that sounds better and better as the model is trained, report statistics on generated music, and let the reader judge the quality by downloading the generated songs.

Controlled time series generation for automotive software-in-the-loop testing using GANs

Abstract: Testing automotive mechatronic systems partly uses the software-in-the-loop approach, where systematically covering inputs of the system-under-test remains a major challenge. In current practice, there are two major techniques of input stimulation. One approach is to craft input sequences which eases control and feedback of the test process but falls short of exposing the system to realistic scenarios. The other is to replay sequences recorded from field operations which accounts for reality but requires collecting a well-labeled dataset of sufficient capacity for widespread use, which is expensive. This work applies the well-known unsupervised learning framework of Generative Adversarial Networks (GAN) to learn an unlabeled dataset of recorded in-vehicle signals and uses it for generation of synthetic input stimuli. Additionally, a metric-based linear interpolation algorithm is demonstrated, which guarantees that generated stimuli follow a customizable similarity relationship with specified references. This combination of techniques enables controlled generation of a rich range of meaningful and realistic input patterns, improving virtual test coverage and reducing the need for expensive field tests.

A list of GAN architectures, their applications, and datasets used in their experiments and evaluation metrics used to judge the quality of the respective GANs. For novel approaches, the GAN name is given as they have been covered already in Section

For a full list of GAN architectures reviewed in this work, their applications, evaluation metrics, and datasets used in their respective experiments, see Table {{table:6c24192c-eff5-45fb-ae2d-fe0845e2a007}} . Results for the sine wave and ECG generation using variants of GAN architectures can be found in Tables {{table:06801933-a4c1-434b-a00f-f5d67e852cf9}}  and {{table:6cde0a85-74e1-4286-b8e0-43f942a1e2d7}} , respectively.


Entangled photons from on-chip slow light

Abstract: We report the first entanglement generation experiment using an on-chip slow light device. With highly efficient spontaneous four-wave mixing enhanced by the slow light effect in a coupled resonator optical waveguide based on a silicon photonic crystal, we generated 1.5-$\mu$m-band high-dimensional time-bin entangled photon pairs. We undertook two-photon interference experiments and observed the coincidence fringes with visibilities $>74\%$. The present result enables us to realize an on-chip entanglement source with a very small footprint, which is an essential function for quantum information processing based on integrated quantum photonics.

Realizing an Entanglement-Based Multiuser Quantum Network with Integrated Photonics

Abstract: Quantum network facilitates the secure transmission of information between different users. Establishing communication links among multiple users in a scalable and efficient way is important for realizing a large-scale quantum network. Here we develop an energy-time entanglement-based dense wavelength division multiplexed network based on an integrated silicon nitride micro-ring resonator, which offers a wide frequency span (covering at least the entire C-band) and narrow bandwidth modes (~ 650MHz). Six pairs of photons are selected to form a fully and simultaneously connected four-user quantum network. The observed quantum interference visibilities are well above the classical limits among all users. Each pair of users perform the BBM92 protocol for quantum key distribution. Our results pave the way for realizing large-scale quantum networks with integrated photonic architecture.

Ultra-bright entangled-photon pair generation from an AlGaAs-on-insulator microring resonator

Abstract: Entangled-photon pairs are an essential resource for quantum information technologies. Chip-scale sources of entangled pairs have been integrated with various photonic platforms, including silicon, nitrides, indium phosphide, and lithium niobate, but each has fundamental limitations that restrict the photon-pair brightness and quality, including weak optical nonlinearity or high waveguide loss. Here, we demonstrate a novel, ultra-low-loss AlGaAs-on-insulator platform capable of generating time-energy entangled photons in a $Q$ $>1$ million microring resonator with nearly 1,000-fold improvement in brightness compared to existing sources. The waveguide-integrated source exhibits an internal generation rate greater than $20\times 10^9$ pairs sec$^{-1}$ mW$^{-2}$, emits near 1550 nm, produces heralded single photons with $>99\%$ purity, and violates Bell's inequality by more than 40 standard deviations with visibility $>97\%$. Combined with the high optical nonlinearity and optical gain of AlGaAs for active component integration, these are all essential features for a scalable quantum photonic platform.

The {{formula:381d95fd-2452-4ad8-b9c6-8b7069e1dd32}}  media for creating energy-time and time-bin entanglement

Four-wave mixing (FWM) is a typical third-order nonlinear optical phenomenon that finds wide applications in wavelength conversion {{cite:8d93fab}}, ultrashort pulse generation {{cite:f1a3c37}}, microscopy {{cite:9501261}}. In some applications, both pump and seeded fields are injected into the {{formula:eb79382c-a2ff-44d4-b0c1-025ad81c6ecb}}  medium, where the FWM process coherently amplifies the seeded fields to a desired power by consuming the pump power {{cite:8d93fab}}. In other applications, only the pump field is injected into a medium placed in a resonator {{cite:5f88e37}}. The FWM process first amplifies the vacuum fluctuations, and the resulting creations, acting as seeds, are further coherently amplified to a significant power level via gain and feedback in the resonator. In these classical applications, the created optical fields are generated by the stimulation and coherent amplification of the seeded fields, which results in considerable power outputs. In 2002, an experiment was conducted for the first time to investigate a completely spontaneous FWM process, where the generated fields arise solely from the parametric amplification of vacuum fluctuations, resulting in power at the single-photon level {{cite:723fecf}}. This process, later known as spontaneous FWM (SFWM), exhibits significant photon correlations. Subsequently, numerous studies have been conducted to create correlated or entangled photons through the SFWM process, utilizing nonresonant {{formula:b557ca39-58f2-4481-ac2a-9ce3f5c300b8}}  nonlinearities in silica fiber {{cite:d5557c4}}, silicon waveguides {{cite:58233ac}}, silicon nitride waveguides {{cite:5110c86}}, and resonant {{formula:9ab39d25-32d6-48a5-b325-d1bbb0c97de2}}  in atomic {{cite:33a05b3}} or molecular ensembles {{cite:847ec94}}. Advancements in integrated optics have enabled SFWM on a chip, making it an effective approach for generating non-classical optical fields. Tab. {{table:d2faed5e-05ad-47b3-b679-8bf09ca4f3ff}}  presents some prominent non-resonant {{formula:ab9a47a2-2546-4921-98e8-d7c80ede9f44}}  materials or structures used for generating energy-time and time-bin entanglement.