NPA: Neural News Recommendation with Personalized Attention

Abstract: News recommendation is very important to help users find interested news and alleviate information overload. Different users usually have different interests and the same user may have various interests. Thus, different users may click the same news article with attention on different aspects. In this paper, we propose a neural news recommendation model with personalized attention (NPA). The core of our approach is a news representation model and a user representation model. In the news representation model we use a CNN network to learn hidden representations of news articles based on their titles. In the user representation model we learn the representations of users based on the representations of their clicked news articles. Since different words and different news articles may have different informativeness for representing news and users, we propose to apply both word- and news-level attention mechanism to help our model attend to important words and news articles. In addition, the same news article and the same word may have different informativeness for different users. Thus, we propose a personalized attention network which exploits the embedding of user ID to generate the query vector for the word- and news-level attentions. Extensive experiments are conducted on a real-world news recommendation dataset collected from MSN news, and the results validate the effectiveness of our approach on news recommendation.

Neural News Recommendation with Negative Feedback

Abstract: News recommendation is important for online news services. Precise user interest modeling is critical for personalized news recommendation. Existing news recommendation methods usually rely on the implicit feedback of users like news clicks to model user interest. However, news click may not necessarily reflect user interests because users may click a news due to the attraction of its title but feel disappointed at its content. The dwell time of news reading is an important clue for user interest modeling, since short reading dwell time usually indicates low and even negative interest. Thus, incorporating the negative feedback inferred from the dwell time of news reading can improve the quality of user modeling. In this paper, we propose a neural news recommendation approach which can incorporate the implicit negative user feedback. We propose to distinguish positive and negative news clicks according to their reading dwell time, and respectively learn user representations from positive and negative news clicks via a combination of Transformer and additive attention network. In addition, we propose to compute a positive click score and a negative click score based on the relevance between candidate news representations and the user representations learned from the positive and negative news clicks. The final click score is a combination of positive and negative click scores. Besides, we propose an interactive news modeling method to consider the relatedness between title and body in news modeling. Extensive experiments on real-world dataset validate that our approach can achieve more accurate user interest modeling for news recommendation.

Two Birds with One Stone: Unified Model Learning for Both Recall and Ranking in News Recommendation

Abstract: Recall and ranking are two critical steps in personalized news recommendation. Most existing news recommender systems conduct personalized news recall and ranking separately with different models. However, maintaining multiple models leads to high computational cost and poses great challenges to meeting the online latency requirement of news recommender systems. In order to handle this problem, in this paper we propose UniRec, a unified method for recall and ranking in news recommendation. In our method, we first infer user embedding for ranking from the historical news click behaviors of a user using a user encoder model. Then we derive the user embedding for recall from the obtained user embedding for ranking by using it as the attention query to select a set of basis user embeddings which encode different general user interests and synthesize them into a user embedding for recall. The extensive experiments on benchmark dataset demonstrate that our method can improve both efficiency and effectiveness for recall and ranking in news recommendation.

FeedRec: News Feed Recommendation with Various User Feedbacks

Abstract: Accurate user interest modeling is important for news recommendation. Most existing methods for news recommendation rely on implicit feedbacks like click for inferring user interests and model training. However, click behaviors usually contain heavy noise, and cannot help infer complicated user interest such as dislike. Besides, the feed recommendation models trained solely on click behaviors cannot optimize other objectives such as user engagement. In this paper, we present a news feed recommendation method that can exploit various kinds of user feedbacks to enhance both user interest modeling and model training. We propose a unified user modeling framework to incorporate various explicit and implicit user feedbacks to infer both positive and negative user interests. In addition, we propose a strong-to-weak attention network that uses the representations of stronger feedbacks to distill positive and negative user interests from implicit weak feedbacks for accurate user interest modeling. Besides, we propose a multi-feedback model training framework to learn an engagement-aware feed recommendation model. Extensive experiments on a real-world dataset show that our approach can effectively improve the model performance in terms of both news clicks and user engagement.

FairRec: Fairness-aware News Recommendation with Decomposed Adversarial Learning

Abstract: News recommendation is important for online news services. Existing news recommendation models are usually learned from users' news click behaviors. Usually the behaviors of users with the same sensitive attributes (e.g., genders) have similar patterns and news recommendation models can easily capture these patterns. It may lead to some biases related to sensitive user attributes in the recommendation results, e.g., always recommending sports news to male users, which is unfair since users may not receive diverse news information. In this paper, we propose a fairness-aware news recommendation approach with decomposed adversarial learning and orthogonality regularization, which can alleviate unfairness in news recommendation brought by the biases of sensitive user attributes. In our approach, we propose to decompose the user interest model into two components. One component aims to learn a bias-aware user embedding that captures the bias information on sensitive user attributes, and the other aims to learn a bias-free user embedding that only encodes attribute-independent user interest information for fairness-aware news recommendation. In addition, we propose to apply an attribute prediction task to the bias-aware user embedding to enhance its ability on bias modeling, and we apply adversarial learning to the bias-free user embedding to remove the bias information from it. Moreover, we propose an orthogonality regularization method to encourage the bias-free user embeddings to be orthogonal to the bias-aware one to better distinguish the bias-free user embedding from the bias-aware one. For fairness-aware news ranking, we only use the bias-free user embedding. Extensive experiments on benchmark dataset show that our approach can effectively improve fairness in news recommendation with minor performance loss.

Empowering News Recommendation with Pre-trained Language Models

Abstract: Personalized news recommendation is an essential technique for online news services. News articles usually contain rich textual content, and accurate news modeling is important for personalized news recommendation. Existing news recommendation methods mainly model news texts based on traditional text modeling methods, which is not optimal for mining the deep semantic information in news texts. Pre-trained language models (PLMs) are powerful for natural language understanding, which has the potential for better news modeling. However, there is no public report that shows PLMs have been applied to news recommendation. In this paper, we report our work on pre-trained language models empowered news recommendation (PLM-NR). Offline experimental results on both monolingual and multilingual news recommendation datasets show that leveraging PLMs for news modeling can effectively improve the performance of news recommendation. Our PLM-NR models have been deployed to the Microsoft News platform, and online flight results show that they can achieve significant performance gains in both English-speaking and global markets.

DKN: Deep Knowledge-Aware Network for News Recommendation

Abstract: Online news recommender systems aim to address the information explosion of news and make personalized recommendation for users. In general, news language is highly condensed, full of knowledge entities and common sense. However, existing methods are unaware of such external knowledge and cannot fully discover latent knowledge-level connections among news. The recommended results for a user are consequently limited to simple patterns and cannot be extended reasonably. Moreover, news recommendation also faces the challenges of high time-sensitivity of news and dynamic diversity of users' interests. To solve the above problems, in this paper, we propose a deep knowledge-aware network (DKN) that incorporates knowledge graph representation into news recommendation. DKN is a content-based deep recommendation framework for click-through rate prediction. The key component of DKN is a multi-channel and word-entity-aligned knowledge-aware convolutional neural network (KCNN) that fuses semantic-level and knowledge-level representations of news. KCNN treats words and entities as multiple channels, and explicitly keeps their alignment relationship during convolution. In addition, to address users' diverse interests, we also design an attention module in DKN to dynamically aggregate a user's history with respect to current candidate news. Through extensive experiments on a real online news platform, we demonstrate that DKN achieves substantial gains over state-of-the-art deep recommendation models. We also validate the efficacy of the usage of knowledge in DKN.

HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation

Abstract: User interest modeling is critical for personalized news recommendation. Existing news recommendation methods usually learn a single user embedding for each user from their previous behaviors to represent their overall interest. However, user interest is usually diverse and multi-grained, which is difficult to be accurately modeled by a single user embedding. In this paper, we propose a news recommendation method with hierarchical user interest modeling, named HieRec. Instead of a single user embedding, in our method each user is represented in a hierarchical interest tree to better capture their diverse and multi-grained interest in news. We use a three-level hierarchy to represent 1) overall user interest; 2) user interest in coarse-grained topics like sports; and 3) user interest in fine-grained topics like football. Moreover, we propose a hierarchical user interest matching framework to match candidate news with different levels of user interest for more accurate user interest targeting. Extensive experiments on two real-world datasets validate our method can effectively improve the performance of user modeling for personalized news recommendation.

Personalized News Recommendation with Knowledge-aware Interactive Matching

Abstract: The most important task in personalized news recommendation is accurate matching between candidate news and user interest. Most of existing news recommendation methods model candidate news from its textual content and user interest from their clicked news in an independent way. However, a news article may cover multiple aspects and entities, and a user usually has different kinds of interest. Independent modeling of candidate news and user interest may lead to inferior matching between news and users. In this paper, we propose a knowledge-aware interactive matching method for news recommendation. Our method interactively models candidate news and user interest to facilitate their accurate matching. We design a knowledge-aware news co-encoder to interactively learn representations for both clicked news and candidate news by capturing their relatedness in both semantic and entities with the help of knowledge graphs. We also design a user-news co-encoder to learn candidate news-aware user interest representation and user-aware candidate news representation for better interest matching. Experiments on two real-world datasets validate that our method can effectively improve the performance of news recommendation.

Neural News Recommendation with Attentive Multi-View Learning

Abstract: Personalized news recommendation is very important for online news platforms to help users find interested news and improve user experience. News and user representation learning is critical for news recommendation. Existing news recommendation methods usually learn these representations based on single news information, e.g., title, which may be insufficient. In this paper we propose a neural news recommendation approach which can learn informative representations of users and news by exploiting different kinds of news information. The core of our approach is a news encoder and a user encoder. In the news encoder we propose an attentive multi-view learning model to learn unified news representations from titles, bodies and topic categories by regarding them as different views of news. In addition, we apply both word-level and view-level attention mechanism to news encoder to select important words and views for learning informative news representations. In the user encoder we learn the representations of users based on their browsed news and apply attention mechanism to select informative news for user representation learning. Extensive experiments on a real-world dataset show our approach can effectively improve the performance of news recommendation.

Contextual Hybrid Session-Based News Recommendation With Recurrent Neural Networks

Abstract: Recommender systems help users deal with information overload by providing tailored item suggestions to them. The recommendation of news is often considered to be challenging, since the relevance of an article for a user can depend on a variety of factors, including the user’s short-term reading interests, the reader’s context, or the recency or popularity of an article. Previous work has shown that the use of Recurrent Neural Networks is promising for the next-in-session prediction task, but has certain limitations when only recorded item click sequences are used as input. In this work, we present a contextual hybrid, deep learning based approach for session-based news recommendation that is able to leverage a variety of information types. We evaluated our approach on two public datasets, using a temporal evaluation protocol that simulates the dynamics of a news portal in a realistic way. Our results confirm the benefits of considering additional types of information, including article popularity and recency, in the proposed way, resulting in significantly higher recommendation accuracy and catalog coverage than other session-based algorithms. Additional experiments show that the proposed parameterizable loss function used in our method also allows us to balance two usually conflicting quality factors, accuracy and novelty.

PP-Rec: News Recommendation with Personalized User Interest and Time-aware News Popularity

Abstract: Personalized news recommendation methods are widely used in online news services. These methods usually recommend news based on the matching between news content and user interest inferred from historical behaviors. However, these methods usually have difficulties in making accurate recommendations to cold-start users, and tend to recommend similar news with those users have read. In general, popular news usually contain important information and can attract users with different interests. Besides, they are usually diverse in content and topic. Thus, in this paper we propose to incorporate news popularity information to alleviate the cold-start and diversity problems for personalized news recommendation. In our method, the ranking score for recommending a candidate news to a target user is the combination of a personalized matching score and a news popularity score. The former is used to capture the personalized user interest in news. The latter is used to measure time-aware popularity of candidate news, which is predicted based on news content, recency, and real-time CTR using a unified framework. Besides, we propose a popularity-aware user encoder to eliminate the popularity bias in user behaviors for accurate interest modeling. Experiments on two real-world datasets show our method can effectively improve the accuracy and diversity for news recommendation.

MM-Rec: Multimodal News Recommendation

Abstract: Accurate news representation is critical for news recommendation. Most of existing news representation methods learn news representations only from news texts while ignore the visual information in news like images. In fact, users may click news not only because of the interest in news titles but also due to the attraction of news images. Thus, images are useful for representing news and predicting user behaviors. In this paper, we propose a multimodal news recommendation method, which can incorporate both textual and visual information of news to learn multimodal news representations. We first extract region-of-interests (ROIs) from news images via object detection. Then we use a pre-trained visiolinguistic model to encode both news texts and news image ROIs and model their inherent relatedness using co-attentional Transformers. In addition, we propose a crossmodal candidate-aware attention network to select relevant historical clicked news for accurate user modeling by measuring the crossmodal relatedness between clicked news and candidate news. Experiments validate that incorporating multimodal news information can effectively improve news recommendation.

DebiasedRec: Bias-aware User Modeling and Click Prediction for Personalized News Recommendation

Abstract: News recommendation is critical for personalized news access. Existing news recommendation methods usually infer users' personal interest based on their historical clicked news, and train the news recommendation models by predicting future news clicks. A core assumption behind these methods is that news click behaviors can indicate user interest. However, in practical scenarios, beyond the relevance between user interest and news content, the news click behaviors may also be affected by other factors, such as the bias of news presentation in the online platform. For example, news with higher positions and larger sizes are usually more likely to be clicked. The bias of clicked news may bring noises to user interest modeling and model training, which may hurt the performance of the news recommendation model. In this paper, we propose a bias-aware personalized news recommendation method named DebiasRec, which can handle the bias information for more accurate user interest inference and model training. The core of our method includes a bias representation module, a bias-aware user modeling module, and a bias-aware click prediction module. The bias representation module is used to model different kinds of news bias and their interactions to capture their joint effect on click behaviors. The bias-aware user modeling module aims to infer users' debiased interest from the clicked news articles by using their bias information to calibrate the interest model. The bias-aware click prediction module is used to train a debiased news recommendation model from the biased click behaviors, where the click score is decomposed into a preference score indicating user's interest in the news content and a news bias score inferred from its different bias features. Experiments on two real-world datasets show that our method can effectively improve the performance of news recommendation.

Graph Neural News Recommendation with Long-term and Short-term Interest Modeling

Abstract: With the information explosion of news articles, personalized news recommendation has become important for users to quickly find news that they are interested in. Existing methods on news recommendation mainly include collaborative filtering methods which rely on direct user-item interactions and content based methods which characterize the content of user reading history. Although these methods have achieved good performances, they still suffer from data sparse problem, since most of them fail to extensively exploit high-order structure information (similar users tend to read similar news articles) in news recommendation systems. In this paper, we propose to build a heterogeneous graph to explicitly model the interactions among users, news and latent topics. The incorporated topic information would help indicate a user's interest and alleviate the sparsity of user-item interactions. Then we take advantage of graph neural networks to learn user and news representations that encode high-order structure information by propagating embeddings over the graph. The learned user embeddings with complete historic user clicks capture the users' long-term interests. We also consider a user's short-term interest using the recent reading history with an attention based LSTM model. Experimental results on real-world datasets show that our proposed model significantly outperforms state-of-the-art methods on news recommendation.

Graph Enhanced Representation Learning for News Recommendation

Abstract: With the explosion of online news, personalized news recommendation becomes increasingly important for online news platforms to help their users find interesting information. Existing news recommendation methods achieve personalization by building accurate news representations from news content and user representations from their direct interactions with news (e.g., click), while ignoring the high-order relatedness between users and news. Here we propose a news recommendation method which can enhance the representation learning of users and news by modeling their relatedness in a graph setting. In our method, users and news are both viewed as nodes in a bipartite graph constructed from historical user click behaviors. For news representations, a transformer architecture is first exploited to build news semantic representations. Then we combine it with the information from neighbor news in the graph via a graph attention network. For user representations, we not only represent users from their historically clicked news, but also attentively incorporate the representations of their neighbor users in the graph. Improved performances on a large-scale real-world dataset validate the effectiveness of our proposed method.

Comparison of different methods on news modeling.

To help better understand the relatedness and differences between the methods reviewed above, we summarize the information and models they used for learning news representations in Table {{table:ec0d8d52-23f1-4026-ae5f-58a582adbefa}} .
Next, we provide several discussions on the aforementioned methods for news modeling.

According to the summarization in Table {{table:ec0d8d52-23f1-4026-ae5f-58a582adbefa}} , we can see that the model architectures used for user representation learning are diverse.
Some methods utilize recurrent neural networks to capture the relatedness of news clicked by users, such as EBNR, DAN and CHAMELEON.
With the great success of Transformer models, many methods also use self-attention or Transformer networks to model the global contexts of user behaviors.
However, these sequential models cannot effectively model the high-order relations between user behaviors, which can provide useful contexts for user interest understanding.
Instead of modeling user behaviors as a sequence, in User-as-Graph each user is modeled as a personalized heterogeneous graph, where the relations between behaviors can be fully modeled.
In addition, several works such as GERL and GNUD use graph neural networks to capture the high-order interactions between users and news on the global user-news graphs, which can also help better understand user interest.
However, the computational cost of these graph-based architectures is usually much heavier than sequential models.


Focuses on energy harvesting in the network

Heterogeneous distributed resources gains benefit from the distributed learning frameworks.

Impact of 6G-based massive IoT for multiple access.

ML and DL at the edge devices.

Reduces extra bandwidth.

Distributed RL provides a viable and effective solution.

Optimizes training time and maximizes the throughput.

Virtualizes cells and controls their power consumption.

Distributed Deep RL is incorporated for harvesting energy.

Provides an optimal topology that fits the capabilities of the system.

Significant rewards from the strongly contributing IoT devices.

Guidelines towards the development of personalized FL.

Realistic PFL approaches are highlighted.

Key performance indicators and their dimensions are summarized.

Joint optimization of channel coding and linear spreading.

Machine Learning for Resource Management in Cellular and IoT Networks: Potentials, Current Solutions, and Open Challenges

Abstract: Internet-of-Things (IoT) refers to a massively heterogeneous network formed through smart devices connected to the Internet. In the wake of disruptive IoT with a huge amount and variety of data, Machine Learning (ML) and Deep Learning (DL) mechanisms will play a pivotal role to bring intelligence to the IoT networks. Among other aspects, ML and DL can play an essential role in addressing the challenges of resource management in large-scale IoT networks. In this article, we conduct a systematic and in-depth survey of the ML- and DL-based resource management mechanisms in cellular wireless and IoT networks. We start with the challenges of resource management in cellular IoT and low-power IoT networks, review the traditional resource management mechanisms for IoT networks, and motivate the use of ML and DL techniques for resource management in these networks. Then, we provide a comprehensive survey of the existing ML- and DL-based resource allocation techniques in wireless IoT networks and also techniques specifically designed for HetNets, MIMO and D2D communications, and NOMA networks. To this end, we also identify the future research directions in using ML and DL for resource allocation and management in IoT networks.

Distributed Reinforcement Learning for Age of Information Minimization in Real-Time IoT Systems

Abstract: In this paper, the problem of minimizing the weighted sum of age of information (AoI) and total energy consumption of Internet of Things (IoT) devices is studied. In the considered model, each IoT device monitors a physical process that follows nonlinear dynamics. As the dynamics of the physical process vary over time, each device should find an optimal sampling frequency to sample the real-time dynamics of the physical system and send sampled information to a base station (BS). Due to limited wireless resources, the BS can only select a subset of devices to transmit their sampled information. Thus, edge devices can cooperatively sample their monitored dynamics based on the local observations and the BS will collect the sampled information from the devices immediately, hence avoiding the additional time and energy used for sampling and information transmission. To this end, it is necessary to jointly optimize the sampling policy of each device and the device selection scheme of the BS so as to accurately monitor the dynamics of the physical process using minimum energy. This problem is formulated as an optimization problem whose goal is to minimize the weighted sum of AoI cost and energy consumption. To solve this problem, we propose a novel distributed reinforcement learning (RL) approach for the sampling policy optimization. The proposed algorithm enables edge devices to cooperatively find the global optimal sampling policy using their own local observations. Given the sampling policy, the device selection scheme can be optimized thus minimizing the weighted sum of AoI and energy consumption of all devices. Simulations with real PM 2.5 pollution data show that the proposed algorithm can reduce the sum of AoI by up to 17.8% and 33.9%, respectively, and the total energy consumption by up to 13.2% and 35.1%, respectively, compared to a conventional deep Q network method and a uniform sampling policy.

Distributed Deep Reinforcement Learning for Functional Split Control in Energy Harvesting Virtualized Small Cells

Abstract: To meet the growing quest for enhanced network capacity, mobile network operators (MNOs) are deploying dense infrastructures of small cells. This, in turn, increases the power consumption of mobile networks, thus impacting the environment. As a result, we have seen a recent trend of powering mobile networks with harvested ambient energy to achieve both environmental and cost benefits. In this paper, we consider a network of virtualized small cells (vSCs) powered by energy harvesters and equipped with rechargeable batteries, which can opportunistically offload baseband (BB) functions to a grid-connected edge server depending on their energy availability. We formulate the corresponding grid energy and traffic drop rate minimization problem, and propose a distributed deep reinforcement learning (DDRL) solution. Coordination among vSCs is enabled via the exchange of battery state information. The evaluation of the network performance in terms of grid energy consumption and traffic drop rate confirms that enabling coordination among the vSCs via knowledge exchange achieves a performance close to the optimal. Numerical results also confirm that the proposed DDRL solution provides higher network performance, better adaptation to the changing environment, and higher cost savings with respect to a tabular multi-agent reinforcement learning (MRL) solution used as a benchmark.

Distributed Learning on Heterogeneous Resource-Constrained Devices

Abstract: We consider a distributed system, consisting of a heterogeneous set of devices, ranging from low-end to high-end. These devices have different profiles, e.g., different energy budgets, or different hardware specifications, determining their capabilities on performing certain learning tasks. We propose the first approach that enables distributed learning in such a heterogeneous system. Applying our approach, each device employs a neural network (NN) with a topology that fits its capabilities; however, part of these NNs share the same topology, so that their parameters can be jointly learned. This differs from current approaches, such as federated learning, which require all devices to employ the same NN, enforcing a trade-off between achievable accuracy and computational overhead of training. We evaluate heterogeneous distributed learning for reinforcement learning (RL) and observe that it greatly improves the achievable reward on more powerful devices, compared to current approaches, while still maintaining a high reward on the weaker devices. We also explore supervised learning, observing similar gains.

Towards Personalized Federated Learning

Abstract: In parallel with the rapid adoption of Artificial Intelligence (AI) empowered by advances in AI research, there have been growing awareness and concerns of data privacy. Recent significant developments in the data regulation landscape have prompted a seismic shift in interest towards privacy-preserving AI. This has contributed to the popularity of Federated Learning (FL), the leading paradigm for the training of machine learning models on data silos in a privacy-preserving manner. In this survey, we explore the domain of Personalized FL (PFL) to address the fundamental challenges of FL on heterogeneous data, a universal characteristic inherent in all real-world datasets. We analyze the key motivations for PFL and present a unique taxonomy of PFL techniques categorized according to the key challenges and personalization strategies in PFL. We highlight their key ideas, challenges and opportunities and envision promising future trajectories of research towards new PFL architectural design, realistic PFL benchmarking, and trustworthy PFL approaches.

NOMA for Next-generation Massive IoT: Performance Potential and Technology Directions

Abstract: Broader applications of the Internet of Things (IoT) are expected in the forthcoming 6G system, although massive IoT is already a key scenario in 5G, predominantly relying on physical layer solutions inherited from 4G LTE and primarily using orthogonal multiple access (OMA). In 6G IoT, supporting a massive number of connections will be required for diverse services of the vertical sectors, prompting fundamental studies on how to improve the spectral efficiency of the system. One of the key enabling technologies is non-orthogonal multiple access (NOMA). This paper consists of two parts. In the first part, finite block length theory and the diversity order of multi-user systems will be used to show the significant potential of NOMA compared to traditional OMA. The supremacy of NOMA over OMA is particularly pronounced for asynchronous contention-based systems relying on imperfect link adaptation, which are commonly assumed for massive IoT systems. To approach these performance bounds, in the second part of the paper, several promising technology directions are proposed for 6G massive IoT, including linear spreading, joint spreading & modulation, multi-user channel coding in the context of various techniques for practical uncoordinated transmissions, cell-free operations, etc., from the perspective of NOMA.

Existing Works on Cutting Edge Developments for Dependable IoT Systems

XAI: Cutting Edge Developments for Dependable IoT Systems

The services provided by the XAI models for dependable IoT systems aim to drive through certain value add-ions. Although upgrading the features of XAI is independent of the timeline of using 6G services and edge XAI structures, these are crucial for XAI Security Enhancement and hinder the developments towards integrating XAI with IoT. This section envisages and summarizes the prospective cutting-edge developments in XAI for IoT systems. A few of the most popular cutting-edge developments are summarized in Table {{table:3f47b58b-e929-43fa-b606-f5c0dfbd6209}} .


Human activity recognition from static and moving camera.

Single-layered and hierarchical approaches for action recognition.

A Review on Video-Based Human Activity Recognition

Abstract: This review article surveys extensively the current progresses made toward video-based human activity recognition. Three aspects for human activity recognition are addressed including core technology, human activity recognition systems, and applications from low-level to high-level representation. In the core technology, three critical processing stages are thoroughly discussed mainly: human object segmentation, feature extraction and representation, activity detection and classification algorithms. In the human activity recognition systems, three main types are mentioned, including single person activity recognition, multiple people interaction and crowd behavior, and abnormal activity recognition. Finally the domains of applications are discussed in detail, specifically, on surveillance environments, entertainment environments and healthcare systems. Our survey, which aims to provide a comprehensive state-of-the-art review of the field, also addresses several challenges associated with these systems and applications. Moreover, in this survey, various applications are discussed in great detail, specifically, a survey on the applications in healthcare monitoring systems.

Advances in Human Action Recognition: A Survey

Abstract: Human action recognition has been an important topic in computer vision due to its many applications such as video surveillance, human machine interaction and video retrieval. One core problem behind these applications is automatically recognizing low-level actions and high-level activities of interest. The former is usually the basis for the latter. This survey gives an overview of the most recent advances in human action recognition during the past several years, following a well-formed taxonomy proposed by a previous survey. From this state-of-the-art survey, researchers can view a panorama of progress in this area for future research.

Summary of previous surveys and their key points ordered by year of publication.

Recently, Cheng et al. {{cite:6799251}} reviewed approaches on human action recognition using an approach-based taxonomy, in which all methodologies are classified into two categories: single-layered approaches and hierarchical approaches. In addition, Vrigkas et al. {{cite:f9cb3b2}} categorized human activity recognition methods into two main categories including “unimodal" and “multimodal". Then, they reviewed classification methods for each of these two categories. The survey of Subetha and Chitrakala {{cite:6b367ef}} mainly focused on human activity recognition and human-object interaction methods. Presti et al. {{cite:4572eeb}} provided a survey of human action recognition based on 3D skeletons, summarizing the main technologies, including both hardware and software for solving the problem of action classification inferred from time series of 3D skeletons. In addition, another survey was presented by Kang and Wildes {{cite:f3b2895}}. It summarized various action recognition and detection algorithms, focused on encoding and classifying features. The latest survey on human action recognition was published in early 2016 by Herath et al. {{cite:63c9acc}}, in which the authors reviewed methods based on hand-crafted features and some deep architectures for recognizing actions. Table {{table:8058b865-4b87-4187-bc17-a0d3f6941d4e}}  summarizes previous surveys on human action and activity recognition published from 1997 to 2017 and reviewed in this paper. The surveys in the literature have shown that the common approaches in human action recognition have focused on using hand-designed local features such as HOG/HOF {{cite:d736548}}, {{cite:fc39069}}, SIFT {{cite:e598081}}, or SURF {{cite:4ce55da}}. In addition, these approaches are also extended for more robustness in video processing such as Cuboids {{cite:1dc4b2d}}, HOG3D {{cite:9231efd}}. To the best of our knowledge, there is no review on human action recognition based on deep learning techniques including comparisons of the performance of deep learning based approaches with traditional methods and with each other. Moreover, deep learning is a rapidly growing field, where novel algorithms appear in very short time duration and change the way of understanding and recognizing actions from visual data. That has prompted us to perform this work. Not only to provide a comparative analysis about the current state of human action recognition using deep learning algorithms, but also to point out the new trends in this field. Our survey will add to the latest reviews on human action recognition in the literature.


Deep Image Compositing

Abstract: Image compositing is a task of combining regions from different images to compose a new image. A common use case is background replacement of portrait images. To obtain high quality composites, professionals typically manually perform multiple editing steps such as segmentation, matting and foreground color decontamination, which is very time consuming even with sophisticated photo editing tools. In this paper, we propose a new method which can automatically generate high-quality image compositing with-out any user input. Our method can be trained end-to-end to optimize exploitation of contextual and color information of both foreground and background images, where the com-positing quality is considered in the optimization. Specifically, inspired by Laplacian pyramid blending, a dense-connected multi-stream fusion network is proposed to effectively fuse the information from the foreground and back-ground images at different scales. In addition, we intro-duce a self-taught strategy to progressively train from easy to complex cases to mitigate the lack of training data. Experiments show that the proposed method can automatically generate high-quality composites and outperforms existing methods both qualitatively and quantitatively.

Deep Image Harmonization

Abstract: Compositing is one of the most common operations in photo editing. To generate realistic composites, the appearances of foreground and background need to be adjusted to make them compatible. Previous approaches to harmonize composites have focused on learning statistical relationships between hand-crafted appearance features of the foreground and background, which is unreliable especially when the contents in the two layers are vastly different. In this work, we propose an end-to-end deep convolutional neural network for image harmonization, which can capture both the context and semantic information of the composite images during harmonization. We also introduce an efficient way to collect large-scale and high-quality training data that can facilitate the training process. Experiments on the synthesized dataset and real composite images show that the proposed network outperforms previous state-of-the-art methods.

GP-GAN: Towards Realistic High-Resolution Image Blending

Abstract: It is common but challenging to address high-resolution image blending in the automatic photo editing application. In this paper, we would like to focus on solving the problem of high-resolution image blending, where the composite images are provided. We propose a framework called Gaussian-Poisson Generative Adversarial Network (GP-GAN) to leverage the strengths of the classical gradient-based approach and Generative Adversarial Networks. To the best of our knowledge, it's the first work that explores the capability of GANs in high-resolution image blending task. Concretely, we propose Gaussian-Poisson Equation to formulate the high-resolution image blending problem, which is a joint optimization constrained by the gradient and color information. Inspired by the prior works, we obtain gradient information via applying gradient filters. To generate the color information, we propose a Blending GAN to learn the mapping between the composite images and the well-blended ones. Compared to the alternative methods, our approach can deliver high-resolution, realistic images with fewer bleedings and unpleasant artifacts. Experiments confirm that our approach achieves the state-of-the-art performance on Transient Attributes dataset. A user study on Amazon Mechanical Turk finds that the majority of workers are in favor of the proposed method. The source code is available in \urlhttps://github.com/wuhuikai/GP-GAN, and there's also an online demo in \urlhttp://wuhuikai.me/DeepJS.

Spatial Fusion GAN for Image Synthesis

Abstract: Recent advances in generative adversarial networks (GANs) have shown great potentials in realistic image synthesis whereas most existing works address synthesis realism in either appearance space or geometry space but few in both. This paper presents an innovative Spatial Fusion GAN (SF-GAN) that combines a geometry synthesizer and an appearance synthesizer to achieve synthesis realism in both geometry and appearance spaces. The geometry synthesizer learns contextual geometries of background images and transforms and places foreground objects into the background images unanimously. The appearance synthesizer adjust the color, brightness and styles of the foreground objects and embeds them into background images harmoniously, where a guided filter is incorporated for detail preserving. The two synthesizers are inter-connected as mutual references which can be trained end-to-end with little supervision. The SF-GAN has been evaluated in two tasks: (1) realistic scene text image synthesis for training better recognition models; (2) glass and hat wearing for realistic matching glasses and hats with real portraits. Qualitative and quantitative comparisons with the state-of-the-art demonstrate the superiority of the proposed SF-GAN.

Modeling Visual Context is Key to Augmenting Object Detection Datasets

Abstract: Performing data augmentation for learning deep neural networks is well known to be important for training visual recognition systems. By artificially increasing the number of training examples, it helps reducing overfitting and improves generalization. For object detection, classical approaches for data augmentation consist of generating images obtained by basic geometrical transformations and color changes of original training images. In this work, we go one step further and leverage segmentation annotations to increase the number of object instances present on training data. For this approach to be successful, we show that modeling appropriately the visual context surrounding objects is crucial to place them in the right environment. Otherwise, we show that the previous strategy actually hurts. With our context model, we achieve significant mean average precision improvements when few labeled examples are available on the VOC'12 benchmark.

The issues to be solved in image composition task and the corresponding deep learning methods to solve these issue. Note that some methods only focus on one issue while some methods attempt to solve multiple issues simultaneously. “Boundary" means refining the boundary between foreground and background. “Appearance" means adjusting the illumination of foreground. “Shadow" means generating shadow for the foreground. “Geometry" means seeking for reasonable location, size, and shape for the foreground considering geometric constraints. “Occlusion" means coping with the unreasonable occlusion. “Semantics" means finding suitable semantic context for the object.

In the following sections, we will first introduce different sub-tasks arranged in the sequential order of creating a composite image. Specifically, as illustrated in Fig. {{figure:d9978d84-a3d4-40f7-a1aa-e21b73ba1950}} , given a pair of foreground and background, we first use
object placement to find suitable scale and location for the foreground (see Section ), and use image blending (see Section ) to refine the boundary between foreground and background. Then, we use image harmonization (see Section ) to adjust the foreground illumination and shadow generation (see Section ) to generate plausible shadow for the foreground. In each section corresponding to one sub-task, we will introduce the traditional methods and deep learning based methods for this sub-task. Because different sub-tasks employ different datasets, we will also review the commonly used datasets and evaluation metrics in each section. Previous works on image composition may focus on one sub-task or multiple sub-tasks, aiming to address one issue or multiple issues. For ease of locating the suitable method to use, we summarize all issues and the corresponding methods to solve them in Table {{table:946a84f4-f69f-4300-a2a1-ae19b2c6ca7b}} .
Furthermore, we will point out some limitations and suggest some potential directions in Section . Finally, we will conclude the whole paper in Section . The contributions of this paper can be summarized as follows,

2) There are many issues to be solved to obtain a realistic composite image. As shown in Table {{table:946a84f4-f69f-4300-a2a1-ae19b2c6ca7b}} , existing methods attempt to solve one issue or multiple issues simultaneously. Since each issue could be very complicated and hard to solve, different research directions (e.g., object placement, image harmonization) diverge from image composition to solve different issues. There is no unified system to address all the issues in image composition, which increases the difficulty in deploying image composition model in real-world applications.


Analysis of recommendation systems for energy efficiency in buildings

Review on building energy management systems focusing on three model types

A survey of recommender systems for energy efficiency in buildings: Principles, challenges and prospects

Abstract: Recommender systems have significantly developed in recent years in parallel with the witnessed advancements in both internet of things (IoT) and artificial intelligence (AI) technologies. Accordingly, as a consequence of IoT and AI, multiple forms of data are incorporated in these systems, e.g. social, implicit, local and personal information, which can help in improving recommender systems' performance and widen their applicability to traverse different disciplines. On the other side, energy efficiency in the building sector is becoming a hot research topic, in which recommender systems play a major role by promoting energy saving behavior and reducing carbon emissions. However, the deployment of the recommendation frameworks in buildings still needs more investigations to identify the current challenges and issues, where their solutions are the keys to enable the pervasiveness of research findings, and therefore, ensure a large-scale adoption of this technology. Accordingly, this paper presents, to the best of the authors' knowledge, the first timely and comprehensive reference for energy-efficiency recommendation systems through (i) surveying existing recommender systems for energy saving in buildings; (ii) discussing their evolution; (iii) providing an original taxonomy of these systems based on specified criteria, including the nature of the recommender engine, its objective, computing platforms, evaluation metrics and incentive measures; and (iv) conducting an in-depth, critical analysis to identify their limitations and unsolved issues. The derived challenges and areas of future implementation could effectively guide the energy research community to improve the energy-efficiency in buildings and reduce the cost of developed recommender systems-based solutions.

Intelligent Systems for Building Energy and Occupant Comfort Optimization: A State of the Art Review and Recommendations

Abstract: Today, buildings consume more than 40% of primary energy in and produce more than 36% of CO2. So, an intelligent controller applied to the buildings for energy and comfort management could achieve significant reduction in energy consumption while improving occupant’s comfort. Conventional on/off controllers were only able to automate the tasks in building and were not well suited for energy optimization tasks. Therefore, building energy management has become a focal point in recent years, promising the development of various technologies for various scenarios. This paper deals with a state of the art review on recent developments in building energy management system (BEMS) and occupants comfort, focusing on three model types: white box, black box, and gray box models. Through a comparative study, this paper presents pros and cons of each model.

Previous Works Related to Intelligent Energy Management Systems

In this section, we present surveys and reviews that are related to IEMS (Table {{table:31f22226-d96d-4696-b7d7-08910fe033cf}} ). Our intention is to provide the reader an extensive look in the field of Energy Management Systems in Residential, Educational and Commercial Buildings. One could read the work of {{cite:6fb278b}} in order to understand which are the general approaches to energy efficiency and also the main architectural, technological and algorithmic aspects of an energy management system. {{cite:12cdc73}} proposed a similar architecture that also incorporates smart appliances, while {{cite:c7fbad5}} presented a more abstract architecture for IoT-based systems, consisting of three layers: a Perception layer, a Network layer and an Application layer. Finally, {{cite:8962e25}} dealt with a review of the state-of-the-art Building Energy Management Systems(BEMS) focusing on three model approaches: White box, Black box and Grey box models. They also performed a comparative analysis of the factors that have the highest impact in energy consumption.


Semi-Supervised StyleGAN for Disentanglement Learning

Abstract: Disentanglement learning is crucial for obtaining disentangled representations and controllable generation. Current disentanglement methods face several inherent limitations: difficulty with high-resolution images, primarily on learning disentangled representations, and non-identifiability due to the unsupervised setting. To alleviate these limitations, we design new architectures and loss functions based on StyleGAN (Karras et al., 2019), for semi-supervised high-resolution disentanglement learning. We create two complex high-resolution synthetic datasets for systematic testing. We investigate the impact of limited supervision and find that using only 0.25%~2.5% of labeled data is sufficient for good disentanglement on both synthetic and real datasets. We propose new metrics to quantify generator controllability, and observe there may exist a crucial trade-off between disentangled representation learning and controllable generation. We also consider semantic fine-grained image editing to achieve better generalization to unseen images.

Deep Learning Face Attributes in the Wild

Abstract: Predicting face attributes in the wild is challenging due to complex face variations. We propose a novel deep learning framework for attribute prediction in the wild. It cascades two CNNs, LNet and ANet, which are fine-tuned jointly with attribute tags, but pre-trained differently. LNet is pre-trained by massive general object categories for face localization, while ANet is pre-trained by massive face identities for attribute prediction. This framework not only outperforms the state-of-the-art with a large margin, but also reveals valuable facts on learning face representation. (1) It shows how the performances of face localization (LNet) and attribute prediction (ANet) can be improved by different pre-training strategies. (2) It reveals that although the filters of LNet are fine-tuned only with image-level attribute tags, their response maps over entire images have strong indication of face locations. This fact enables training LNet for face localization with only image-level annotations, but without face bounding boxes or landmarks, which are required by all attribute recognition works. (3) It also demonstrates that the high-level hidden neurons of ANet automatically discover semantic concepts after pre-training with massive face identities, and such concepts are significantly enriched after fine-tuning with attribute tags. Each attribute can be well explained with a sparse linear combination of these concepts.

Summary statistics of the experimental datasets.

Table {{table:d4978f9c-6f9e-4d7d-bb0a-75d8e883c73b}}  summarizes the basic statistics of all 4 datasets used in our experiments. Note that #Factors = dimension of label {{formula:3c9c28d4-6769-4d4e-a9ca-0fc8b1fa6853}}  and #latent = dimension of label {{formula:67559a79-1895-4b5c-a870-fe291c05dd3b}}  + dimension of nuisance {{formula:145249ec-37ce-475c-be1f-9f0f03e513e0}} . For the CelebA dataset, we additionally crop and align so each image contain only the face without background and resize to 64{{formula:87809c13-0630-4974-befa-79a27a4a2455}} 64 by simple downsampling. For the Isaac3D dataset, we resize each image to 64{{formula:949b7c5b-7824-41d8-9198-0332a5e9d0f3}} 64 with a bilinear downsampling.


1e-2 for addition and 5e-3 for all other operations

Neural Arithmetic Units

Abstract: Neural networks can approximate complex functions, but they struggle to perform exact arithmetic operations over real numbers. The lack of inductive bias for arithmetic operations leaves neural networks without the underlying logic necessary to extrapolate on tasks such as addition, subtraction, and multiplication. We present two new neural network components: the Neural Addition Unit (NAU), which can learn exact addition and subtraction; and the Neural Multiplication Unit (NMU) that can multiply subsets of a vector. The NMU is, to our knowledge, the first arithmetic neural network component that can learn to multiply elements from a vector, when the hidden size is large. The two new components draw inspiration from a theoretical analysis of recently proposed arithmetic components. We find that careful initialization, restricting parameter space, and regularizing for sparsity is important when optimizing the NAU and NMU. Our proposed units NAU and NMU, compared with previous neural units, converge more consistently, have fewer parameters, learn faster, can converge for larger hidden sizes, obtain sparse and meaningful weights, and can extrapolate to negative and small values.

Neural Power Units

Abstract: Conventional Neural Networks can approximate simple arithmetic operations, but fail to generalize beyond the range of numbers that were seen during training. Neural Arithmetic Units aim to overcome this difficulty, but current arithmetic units are either limited to operate on positive numbers or can only represent a subset of arithmetic operations. We introduce the Neural Power Unit (NPU) that operates on the full domain of real numbers and is capable of learning arbitrary power functions in a single layer. The NPU thus fixes the shortcomings of existing arithmetic units and extends their expressivity. We achieve this by using complex arithmetic without requiring a conversion of the network to complex numbers. A simplification of the unit to the RealNPU yields a highly interpretable model. We show that the NPUs outperform their competitors in terms of accuracy and sparsity on artificial arithmetic datasets, and that the RealNPU can discover the governing equations of a dynamical systems only from data.

Differences in the`Large Scale Arithmetic' task used in the papers {{cite:058053b}} and {{cite:7b817b5}}. `a' and `b' represent summed slices of the input, and are the expected output values for the addition unit.

iNALU's experiments 4 (`Influence of Initalization') and 5 (`Simple Function Learning Task') is a copy of the task but is different to the original. Experiment 4 calculates a and b differently to {{cite:058053b}} by not allowing for overlap between the slices which form a and b, and experiment 5 use different interpolation and extrapolation ranges to the original experiments.
{{cite:7b817b5}}'s claims that their `Large Scale Arithmetic' task is equivalent to the Arithmetic Dataset task. However, as shown in Table {{table:2fca94a4-e12e-446a-825b-a9815a07d4ca}}  there are key distinctions between the two meaning the results from the two papers are not directly comparable.


Neural Architectures for Fine-grained Entity Type Classification

Abstract: In this work, we investigate several neural network architectures for fine-grained entity type classification. Particularly, we consider extensions to a recently proposed attentive neural architecture and make three key contributions. Previous work on attentive neural architectures do not consider hand-crafted features, we combine learnt and hand-crafted features and observe that they complement each other. Additionally, through quantitative analysis we establish that the attention mechanism is capable of learning to attend over syntactic heads and the phrase containing the mention, where both are known strong hand-crafted features for our task. We enable parameter sharing through a hierarchical label encoding method, that in low-dimensional projections show clear clusters for each type hierarchy. Lastly, despite using the same evaluation dataset, the literature frequently compare models trained using different data. We establish that the choice of training data has a drastic impact on performance, with decreases by as much as 9.85% loose micro F1 score for a previously proposed method. Despite this, our best model achieves state-of-the-art results with 75.36% loose micro F1 score on the well- established FIGER (GOLD) dataset.

Ultra-Fine Entity Typing

Abstract: We introduce a new entity typing task: given a sentence with an entity mention, the goal is to predict a set of free-form phrases (e.g. skyscraper, songwriter, or criminal) that describe appropriate types for the target entity. This formulation allows us to use a new type of distant supervision at large scale: head words, which indicate the type of the noun phrases they appear in. We show that these ultra-fine types can be crowd-sourced, and introduce new evaluation sets that are much more diverse and fine-grained than existing benchmarks. We present a model that can predict ultra-fine types, and is trained using a multitask objective that pools our new head-word supervision with prior supervision from entity linking. Experimental results demonstrate that our model is effective in predicting entity types at varying granularity; it achieves state of the art performance on an existing fine-grained entity typing benchmark, and sets baselines for our newly-introduced datasets.

A Chinese Corpus for Fine-grained Entity Typing

Abstract: Fine-grained entity typing is a challenging task with wide applications. However, most existing datasets for this task are in English. In this paper, we introduce a corpus for Chinese fine-grained entity typing that contains 4,800 mentions manually labeled through crowdsourcing. Each mention is annotated with free-form entity types. To make our dataset useful in more possible scenarios, we also categorize all the fine-grained types into 10 general types. Finally, we conduct experiments with some neural models whose structures are typical in fine-grained entity typing and show how well they perform on our dataset. We also show the possibility of improving Chinese fine-grained entity typing through cross-lingual transfer learning.

The statistic of different entity typing datasets.

To demonstrate that GET is superior to the classification-based approaches, we collect the human-annotated data from four different entity type datasets.
The statistics of them are listed in Table {{table:5586b8ee-f323-48f5-894c-43b463258edd}} .
We compare our model with baselines on BNN, FIGER and Ultra-fine to demonstrate the superior performance of GET on entity typing.
GT dataset is used to evaluate the effectiveness of CL upon the texts of different languages and heterogeneous data.
Please note that we do sample the test set to reduce the cost of manual evaluation on assessing whether the newly-generated types are correct.
However, the results from the baselines and our model are evaluated in the same test instances.


GPT3-to-plan: Extracting plans from text using GPT-3

Abstract: Operations in many essential industries including finance and banking are often characterized by the need to perform repetitive sequential tasks. Despite their criticality to the business, workflows are rarely fully automated or even formally specified, though there may exist a number of natural language documents describing these procedures for the employees of the company. Plan extraction methods provide us with the possibility of extracting structure plans from such natural language descriptions of the plans/workflows, which could then be leveraged by an automated system. In this paper, we investigate the utility of generalized language models in performing such extractions directly from such texts. Such models have already been shown to be quite effective in multiple translation tasks, and our initial results seem to point to their effectiveness also in the context of plan extractions. Particularly, we show that GPT-3 is able to generate plan extraction results that are comparable to many of the current state of the art plan extraction methods.

Large Language Models as Zero-Shot Human Models for Human-Robot Interaction

Abstract: Human models play a crucial role in human-robot interaction (HRI), enabling robots to consider the impact of their actions on people and plan their behavior accordingly. However, crafting good human models is challenging; capturing context-dependent human behavior requires significant prior knowledge and/or large amounts of interaction data, both of which are difficult to obtain. In this work, we explore the potential of large language models (LLMs) — which have consumed vast amounts of human-generated text data — to act as zero-shot human models for HRI. Our experiments on three social datasets yield promising results; the LLMs are able to achieve performance comparable to purpose-built models. That said, we also discuss current limitations, such as sensitivity to prompts and spatial/numerical reasoning mishaps. Based on our findings, we demonstrate how LLM-based human models can be integrated into a social robot's planning process and applied in HRI scenarios focused on the important element of trust. Specifically, we present one case study on a simulated trust-based table-clearing task and replicate past results that relied on custom models. Next, we conduct a new robot utensil-passing experiment ($n=65$) where preliminary results show that planning with an LLM-based human model can achieve gains over a basic myopic plan. In summary, our results show that LLMs offer a promising (but incomplete) approach to human modeling for HRI.

Translating Natural Language to Planning Goals with Large-Language Models

Abstract: Recent large language models (LLMs) have demonstrated remarkable performance on a variety of natural language processing (NLP) tasks, leading to intense excitement about their applicability across various domains. Unfortunately, recent work has also shown that LLMs are unable to perform accurate reasoning nor solve planning problems, which may limit their usefulness for robotics-related tasks. In this work, our central question is whether LLMs are able to translate goals specified in natural language to a structured planning language. If so, LLM can act as a natural interface between the planner and human users; the translated goal can be handed to domain-independent AI planners that are very effective at planning. Our empirical results on GPT 3.5 variants show that LLMs are much better suited towards translation rather than planning. We find that LLMs are able to leverage commonsense knowledge and reasoning to furnish missing details from under-specified goals (as is often the case in natural language). However, our experiments also reveal that LLMs can fail to generate goals in tasks that involve numerical or physical (e.g., spatial) reasoning, and that LLMs are sensitive to the prompts used. As such, these models are promising for translation to structured planning languages, but care should be taken in their use.

Summary of research works exploring GLLMs for planning. Here ZS represents zero-shot, and FS represents few-shot.

Research works exploring GLLMs for planning. Table {{table:2cd7bb61-9a27-44ea-ab40-f29603215e99}}  presents a summary of research works exploring GLLMs for planning. Human models are crucial in facilitating human-robot interaction (HRI), as they empower robots to plan their behaviour based on the impact of their actions on individuals. As it is difficult to craft good human labels, Zhang et al. {{cite:ca9427f}} used the GPT-3.5 model (i) as zero-shot human models and also (ii) for planning in trust-related scenarios. Hu et al. {{cite:c45a8dc}} proposed a novel prompting strategy called “Chain of Symbol” prompting to elicit better the planning abilities of large language models like InstructGPT and ChatGPT. Unlike CoT prompting, which uses natural language descriptions to represent complex environments, CoS prompting uses condensed symbols to represent them in intermediate reasoning steps. The authors reported that CoS prompting outperforms CoT prompting in both performance and efficiency.


Aggregate predictions from an ensemble of multiple models and architectures. The models in the ensemble are trained and optimized separately under different optimization configurations and loss functions.

Imaging features are extracted using a larger encoder, and the segmentation mask is reconstructed using a smaller decoder. A variational auto-encoder branch is added for regularization purposes. The ensemble of 10 models yielded the best performance.

Incorporate several modifications into the nnU-Net architecture {{cite:36aa8b1}}, including heavy data augmentation, region-based training, and post-processing.

Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation

Abstract: Deep learning approaches such as convolutional neural nets have consistently outperformed previous methods on challenging tasks such as dense, semantic segmentation. However, the various proposed networks perform differently, with behaviour largely influenced by architectural choices and training settings. This paper explores Ensembles of Multiple Models and Architectures (EMMA) for robust performance through aggregation of predictions from a wide range of methods. The approach reduces the influence of the meta-parameters of individual models and the risk of overfitting the configuration to a particular database. EMMA can be seen as an unbiased, generic deep learning model which is shown to yield excellent performance, winning the first position in the BRATS 2017 competition among 50+ participating teams.

3D MRI brain tumor segmentation using autoencoder regularization

Abstract: Automated segmentation of brain tumors from 3D magnetic resonance images (MRIs) is necessary for the diagnosis, monitoring, and treatment planning of the disease. Manual delineation practices require anatomical knowledge, are expensive, time consuming and can be inaccurate due to human error. Here, we describe a semantic segmentation network for tumor subregion segmentation from 3D MRIs based on encoder-decoder architecture. Due to a limited training dataset size, a variational auto-encoder branch is added to reconstruct the input image itself in order to regularize the shared decoder and impose additional constraints on its layers. The current approach won 1st place in the BraTS 2018 challenge.

nnU-Net for Brain Tumor Segmentation

Abstract: We apply nnU-Net to the segmentation task of the BraTS 2020 challenge. The unmodified nnU-Net baseline configuration already achieves a respectable result. By incorporating BraTS-specific modifications regarding postprocessing, region-based training, a more aggressive data augmentation as well as several minor modifications to the nnUNet pipeline we are able to improve its segmentation performance substantially. We furthermore re-implement the BraTS ranking scheme to determine which of our nnU-Net variants best fits the requirements imposed by it. Our final ensemble took the first place in the BraTS 2020 competition with Dice scores of 88.95, 85.06 and 82.03 and HD95 values of 8.498,17.337 and 17.805 for whole tumor, tumor core and enhancing tumor, respectively.

3D Deep Learning models used in the winning entries of brain tumor segmentation (BRATS) challenge within the past 5 years

Biomedical Image Segmentation with U-Net

Efficient DNN-based medical image segmentation is often driven by the recent progress in semantic segmentation. In {{cite:b820078}}, the authors employed a network architecture similar to 3D U-Net for bone and brain tumor segmentation. They combined multiple segmentation maps created at different scales, which was shown to speed up convergence. However, this approach can be computationally expensive as it employed large receptive fields in convolutional layers. In {{cite:0407fb2}}, the authors implemented a 3D U-Net that has been modified with respect to up-sampling pathways, normalization methods, number of filters, and batch size, enabling training with large image patches and capturing contextual information that leads to segmentation performance improvements. In a more recent work {{cite:17c9f5a}} a separable 3D U-Net consisting of separable 3D convolutions was proposed. Their S3D-UNet architecture fully utilizes the 3D volumes via the use of several 3D U-Net blocks. Isensee et al. {{cite:cda36ae}} argue that a well-designed U-Net architecture is competent in generating state-of-the-art segmentation maps without the need for significant architectural modifications and that enhancements proposed in prior literature provide no additional benefit. It should be highlighted that the winning contributions of the BRATS challenge in 2019 {{cite:d66dcf6}} and 2020 {{cite:bb260f7}} have also adopted U-Net based architectures. The authors in {{cite:d66dcf6}} utilized a two-stage cascaded U-Net, and Isensee et al. {{cite:bb260f7}} employed the nnU-Net architecture {{cite:36aa8b1}}, initially developed as a general-purpose network for segmentation following a U-Net. 3D Deep Learning models used in the winning entries of the BRATS challenge within the past five years is summarized in Table {{table:1225fa18-3ceb-4909-846a-c2c936b4e082}} .


COVID-19 Radiography Database - Version 3

COVID-19 Image Data Collection: Prospective Predictions Are the Future

Abstract: Across the world's coronavirus disease 2019 (COVID-19) hot spots, the need to streamline patient diagnosis and management has become more pressing than ever. As one of the main imaging tools, chest X-rays (CXRs) are common, fast, non-invasive, relatively cheap, and potentially bedside to monitor the progression of the disease. This paper describes the first public COVID-19 image data collection as well as a preliminary exploration of possible use cases for the data. This dataset currently contains hundreds of frontal view X-rays and is the largest public resource for COVID-19 image and prognostic data, making it a necessary resource to develop and evaluate tools to aid in the treatment of COVID-19. It was manually aggregated from publication figures as well as various web based repositories into a machine learning (ML) friendly format with accompanying dataloader code. We collected frontal and lateral view imagery and metadata such as the time since first symptoms, intensive care unit (ICU) status, survival status, intubation status, or hospital location. We present multiple possible use cases for the data such as predicting the need for the ICU, predicting patient survival, and understanding a patient's trajectory during treatment. Data can be accessed here: https://github.com/ieee8023/covid-chestxray-dataset

Can AI Help in Screening Viral and COVID-19 Pneumonia?

Abstract: Coronavirus disease (COVID-19) is a pandemic disease, which has already caused thousands of causalities and infected several millions of people worldwide. Any technological tool enabling rapid screening of the COVID-19 infection with high accuracy can be crucially helpful to the healthcare professionals. The main clinical tool currently in use for the diagnosis of COVID-19 is the Reverse transcription polymerase chain reaction (RT-PCR), which is expensive, less-sensitive and requires specialized medical personnel. X-ray imaging is an easily accessible tool that can be an excellent alternative in the COVID-19 diagnosis. This research was taken to investigate the utility of artificial intelligence (AI) in the rapid and accurate detection of COVID-19 from chest X-ray images. The aim of this paper is to propose a robust technique for automatic detection of COVID-19 pneumonia from digital chest X-ray images applying pre-trained deep-learning algorithms while maximizing the detection accuracy. A public database was created by the authors combining several public databases and also by collecting images from recently published articles. The database contains a mixture of 423 COVID-19, 1485 viral pneumonia, and 1579 normal chest X-ray images. Transfer learning technique was used with the help of image augmentation to train and validate several pre-trained deep Convolutional Neural Networks (CNNs). The networks were trained to classify two different schemes: i) normal and COVID-19 pneumonia; ii) normal, viral and COVID-19 pneumonia with and without image augmentation. The classification accuracy, precision, sensitivity, and specificity for both the schemes were 99.7%, 99.7%, 99.7% and 99.55% and 97.9%, 97.95%, 97.9%, and 98.8%, respectively. The high accuracy of this computer-aided diagnostic tool can significantly improve the speed and accuracy of COVID-19 diagnosis. This would be extremely useful in this pandemic where disease burden and need for preventive measures are at odds with available resources.

List of datasets that compose the COVIDx8B benchmark dataset.

Most of the early research regarding COVID-19 detection on CXR images suffered from the lack of available datasets {{cite:8068d13}}. The authors would frequently combine different smaller datasets, so fairly comparing the results was impossible. COVIDx8B is a large and heterogeneous COVID-19 CXR benchmark dataset with {{formula:c2f0e791-0f20-4c00-8c07-1cb0c51e3570}}  CXR images coming from patients of at least 51 countries {{cite:53dcaa2}}. It is constructed with images extracted from six open-source chest radiography datasets, which are shown in Table {{table:f84a02a3-8a65-40bf-b2b5-3897018c3d93}} . Notice that the sum of the images in the source datasets is much larger than the size of COVIDx8B since not all of them were selected by the authors. Example images from the COVIDx8B dataset are shown in Figure {{figure:750f3795-c530-4b91-ad26-2e9c7e7f6ef7}} .


Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Abstract: State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available

R-FCN: Object Detection via Region-based Fully Convolutional Networks

Abstract: We present region-based, fully convolutional networks for accurate and efficient object detection. In contrast to previous region-based detectors such as Fast/Faster R-CNN that apply a costly per-region subnetwork hundreds of times, our region-based detector is fully convolutional with almost all computation shared on the entire image. To achieve this goal, we propose position-sensitive score maps to address a dilemma between translation-invariance in image classification and translation-variance in object detection. Our method can thus naturally adopt fully convolutional image classifier backbones, such as the latest Residual Networks (ResNets), for object detection. We show competitive results on the PASCAL VOC datasets (e.g., 83.6% mAP on the 2007 set) with the 101-layer ResNet. Meanwhile, our result is achieved at a test-time speed of 170ms per image, 2.5-20 times faster than the Faster R-CNN counterpart. Code is made publicly available at: https://github.com/daijifeng001/r-fcn.

Searching for MobileNetV3

Abstract: We present the next generation of MobileNets based on a combination of complementary search techniques as well as a novel architecture design. MobileNetV3 is tuned to mobile phone CPUs through a combination of hardware-aware network architecture search (NAS) complemented by the NetAdapt algorithm and then subsequently improved through novel architecture advances. This paper starts the exploration of how automated search algorithms and network design can work together to harness complementary approaches improving the overall state of the art. Through this process we create two new MobileNet models for release: MobileNetV3-Large and MobileNetV3-Small which are targeted for high and low resource use cases. These models are then adapted and applied to the tasks of object detection and semantic segmentation. For the task of semantic segmentation (or any dense pixel prediction), we propose a new efficient segmentation decoder Lite Reduced Atrous Spatial Pyramid Pooling (LR-ASPP). We achieve new state of the art results for mobile classification, detection and segmentation. MobileNetV3-Large is 3.2% more accurate on ImageNet classification while reducing latency by 20% compared to MobileNetV2. MobileNetV3-Small is 6.6% more accurate compared to a MobileNetV2 model with comparable latency. MobileNetV3-Large detection is over 25% faster at roughly the same accuracy as MobileNetV2 on COCO detection. MobileNetV3-Large LR-ASPP is 34% faster than MobileNetV2 R-ASPP at similar accuracy for Cityscapes segmentation.

Objects as Points

Abstract: Detection identifies objects as axis-aligned boxes in an image. Most successful object detectors enumerate a nearly exhaustive list of potential object locations and classify each. This is wasteful, inefficient, and requires additional post-processing. In this paper, we take a different approach. We model an object as a single point --- the center point of its bounding box. Our detector uses keypoint estimation to find center points and regresses to all other object properties, such as size, 3D location, orientation, and even pose. Our center point based approach, CenterNet, is end-to-end differentiable, simpler, faster, and more accurate than corresponding bounding box based detectors. CenterNet achieves the best speed-accuracy trade-off on the MS COCO dataset, with 28.1% AP at 142 FPS, 37.4% AP at 52 FPS, and 45.1% AP with multi-scale testing at 1.4 FPS. We use the same approach to estimate 3D bounding box in the KITTI benchmark and human pose on the COCO keypoint dataset. Our method performs competitively with sophisticated multi-stage methods and runs in real-time.

Overview of Pedestrian Detection Models.

Performance characterization was carried out on seven popular pedestrian detection models. All models use publicly available pretrained weights from two popular model zoos {{cite:5dcb4d9}}{{cite:12a751d}} and are trained using the COCO “train 2017” dataset {{cite:1b25b32}}. An overview of the pedestrian detection models can be seen in Table {{table:e6fd8dbe-f2fa-493c-8ffb-cee7322295fc}} .
The pedestrian detection models chosen for characterization can be divided into 3 categories: Two-Stage Frameworks, One-Stage Frameworks and Keypoint Estimation.
Two-stage frameworks such as FasterRCNN{{cite:f4f3291}}, MaskRCNN{{cite:92ae06d}} and R-FCN{{cite:00d66d2}} apply two separate networks to perform classification. One network is used to propose regions of interest and a dedicated second network performs object detection {{cite:ee5e1a9}}.
One-stage frameworks such as RetinaNet{{cite:d461b45}}, SSD{{cite:853008e}} and SSDLite{{cite:0f3642c}}{{cite:b7c0ec1}} attempt to reduce computation and increase speed by performing object detection using a single feed forward convolutional network that does not interact with a region proposal module. RetinaNet also implements a novel method of “focal loss” which is used to reduce the imbalance between foreground and background classes during training with a view to increasing detection precision.
CenterNet{{cite:a2a7f81}} takes an alternative approach based on keypoint estimation. Objects are represented as a single point at their bounding box center identified by a heat map generated using a fully convolutional network. Other object features such as object size, orientation and pose are then regressed directly from the image features at the center location.


A feature selection algorithm based on a minimum spanning tree is proposed to find the optimal feature set.

Use the sure independence screening (SIS) method. Multiple features are removed in each iteration.

An fMRI Feature Selection Method Based on a Minimum Spanning Tree for Identifying Patients with Autism

Abstract: Autism spectrum disorder (ASD) is a neurodevelopmental disorder originating in infancy and childhood that may cause language barriers and social difficulties. However, in the diagnosis of ASD, the current machine learning methods still face many challenges in determining the location of biomarkers. Here, we proposed a novel feature selection method based on the minimum spanning tree (MST) to seek neuromarkers for ASD. First, we constructed an undirected graph with nodes of candidate features. At the same time, a weight calculation method considering both feature redundancy and discriminant ability was introduced. Second, we utilized the Prim algorithm to construct the MST from the initial graph structure. Third, the sum of the edge weights of all connected nodes was sorted for each node in the MST. Then, N features corresponding to the nodes with the first N smallest sum were selected as classification features. Finally, the support vector machine (SVM) algorithm was used to evaluate the discriminant performance of the aforementioned feature selection method. Comparative experiments results show that our proposed method has improved the ASD classification performance, i.e., the accuracy, sensitivity, and specificity were 86.7%, 87.5%, and 85.7%, respectively.

Prediction of treatment outcome for autism from structure of the brain based on sure independence screening

Abstract: Autism spectrum disorder (ASD) is a complex neurodevelopmental disorder, and behavioral treatment interventions have shown promise for young children with ASD. However, there is limited progress in understanding the effect of each type of treatment. In this project, we aim to detect structural changes in the brain after treatment and select structural features associated with treatment outcomes. The difficulty in building large databases of patients who have received specific treatments and the high dimensionality of medical image analysis problems are the challenges in this work. To select predictive features and build accurate models, we use the sure independence screening (SIS) method. SIS is a theoretically and empirically validated method for ultra-high dimensional general linear models, and it achieves both predictive accuracy and correct feature selection by iterative feature selection. Compared with step-wise feature selection methods, SIS removes multiple features in each iteration and is computationally efficient. Compared with other linear models such as elastic-net regression, support vector regression (SVR) and partial least squares regression (PSLR), SIS achieves higher accuracy. We validated the superior performance of SIS in various experiments: First, we extract brain structural features from FreeSurfer, including cortical thickness, surface area, mean curvature and cortical volume. Next, we predict different measures of treatment outcomes based on structural features. We show that SIS achieves the highest correlation between prediction and measurements in all tasks. Furthermore, we report regions selected by SIS as biomarkers for ASD.

Summary of feature selection techniques related to ASD.

Table {{table:e40f4125-7f40-4bbd-b3f2-dd10467c3092}}  reports the techniques used recently for feature selection in ASD studies. For example, wrapper methods are generally more used comparing to filtering and embedding. As expected that the predictive feature derived WM is higher performance than FM {{cite:2bf5f1f}}. Due to the difficulty of setting parameters, the use of EM is limited. More details about feature selection techniques are reported in {{cite:9d6eaef}}.


FMore: An Incentive Scheme of Multi-dimensional Auction for Federated Learning in MEC

Abstract: Promising federated learning coupled with Mobile Edge Computing (MEC) is considered as one of the most promising solutions to the AI-driven service provision. Plenty of studies focus on federated learning from the performance and security aspects, but they neglect the incentive mechanism. In MEC, edge nodes would not like to voluntarily participate in learning, and they differ in the provision of multi-dimensional resources, both of which might deteriorate the performance of federated learning. Also, lightweight schemes appeal to edge nodes in MEC. These features require the incentive mechanism to be well designed for MEC. In this paper, we present an incentive mechanism FMore with multi-dimensional procurement auction of K winners. Our proposal FMore not only is lightweight and incentive compatible, but also encourages more high-quality edge nodes with low cost to participate in learning and eventually improve the performance of federated learning. We also present theoretical results of Nash equilibrium strategy to edge nodes and employ the expected utility theory to provide guidance to the aggregator. Both extensive simulations and real-world experiments demonstrate that the proposed scheme can effectively reduce the training rounds and drastically improve the model accuracy for challenging AI tasks.

Incentivizing Collaboration in Machine Learning via Synthetic Data Rewards

Abstract: This paper presents a novel collaborative generative modeling (CGM) framework that incentivizes collaboration among self-interested parties to contribute data to a pool for training a generative model (e.g., GAN), from which synthetic data are drawn and distributed to the parties as rewards commensurate to their contributions. Distributing synthetic data as rewards (instead of trained models or money) offers task- and model-agnostic benefits for downstream learning tasks and is less likely to violate data privacy regulation. To realize the framework, we firstly propose a data valuation function using maximum mean discrepancy (MMD) that values data based on its quantity and quality in terms of its closeness to the true data distribution and provide theoretical results guiding the kernel choice in our MMD-based data valuation function. Then, we formulate the reward scheme as a linear optimization problem that when solved, guarantees certain incentives such as fairness in the CGM framework. We devise a weighted sampling algorithm for generating synthetic data to be distributed to each party as reward such that the value of its data and the synthetic data combined matches its assigned reward value by the reward scheme. We empirically show using simulated and real-world datasets that the parties' synthetic data rewards are commensurate to their contributions.

iDML: Incentivized Decentralized Machine Learning

Abstract: With the rising emergence of decentralized and opportunistic approaches to machine learning, end devices are increasingly tasked with training deep learning models on-devices using crowd-sourced data that they collect themselves. These approaches are desirable from a resource consumption perspective and also from a privacy preservation perspective. When the devices benefit directly from the trained models, the incentives are implicit - contributing devices' resources are incentivized by the availability of the higher-accuracy model that results from collaboration. However, explicit incentive mechanisms must be provided when end-user devices are asked to contribute their resources (e.g., computation, communication, and data) to a task performed primarily for the benefit of others, e.g., training a model for a task that a neighbor device needs but the device owner is uninterested in. In this project, we propose a novel blockchain-based incentive mechanism for completely decentralized and opportunistic learning architectures. We leverage a smart contract not only for providing explicit incentives to end devices to participate in decentralized learning but also to create a fully decentralized mechanism to inspect and reflect on the behavior of the learning architecture.

Summary of the most related literature on incentive-based collaboration.

Our proposed ICL framework builds on the related literature on incentivized collaborative systems, which often focuses on one or more of three components: pricing, payment allocation, and participant selection {{cite:dd45afd}}. Many incentive mechanisms have been designed for these components, depending on particular application scenarios. In Table {{table:581cdc91-32de-4e81-91a6-edca5d822cb6}} , we summarize a few representative works and their main features.
Next, we will relate these existing works to our proposed ICL framework.