Dynamic RIS configuration with NOMA is the capacity-achieving transmission strategy

Asymmetric and symmetric deployment strategies are preferred by NOMA and OMA, respectively

Signal cancelation based RIS phase shift design

Multiple RISs for cell-edge users to extend coverage

Simultaneous intra-cell signal enhancement and inter-cell interference mitigation for A2G communications

Capacity and Optimal Resource Allocation for IRS-Assisted Multi-User Communication Systems

Abstract: The fundamental capacity limits of intelligent reflecting surface (IRS)-assisted multi-user wireless communication systems are investigated in this article. Specifically, the capacity and rate regions for both capacity-achieving non-orthogonal multiple access (NOMA) and orthogonal multiple access (OMA) transmission schemes are characterized by jointly optimizing the IRS reflection matrix and wireless resource allocation under the constraints of a maximum number of IRS reconfiguration times. In NOMA, all users are served in the same resource blocks by employing superposition coding and successive interference cancelation techniques. In OMA, all users are served by being allocated orthogonal resource blocks of different sizes. For NOMA, the ideal case with an asymptotically large number of IRS reconfiguration times is firstly considered, where the optimal solution is obtained by employing the Lagrange duality method. Inspired by this result, an inner bound of the capacity region for the general case with a finite number of IRS reconfiguration times is derived. For OMA, the optimal transmission strategy for the ideal case is to serve each individual user alternatingly with its effective channel power gain maximized. Based on this result, a rate region inner bound for the general case is derived. Finally, numerical results are provided to show that: i) a significant capacity and rate region improvement can be achieved by using IRS; ii) the capacity gain can be further improved by dynamically configuring the IRS reflection matrix.

Joint Deployment and Multiple Access Design for Intelligent Reflecting Surface Assisted Networks

Abstract: The fundamental intelligent reflecting surface (IRS) deployment problem is investigated for IRS-assisted networks, where one IRS is arranged to be deployed in a specific region for assisting the communication between an access point (AP) and multiple users. Specifically, three multiple access schemes are considered, namely non-orthogonal multiple access (NOMA), frequency division multiple access (FDMA), and time division multiple access (TDMA). The weighted sum rate maximization problem for joint optimization of the deployment location and the reflection coefficients of the IRS as well as the power allocation at the AP is formulated. The non-convex optimization problems obtained for NOMA and FDMA are solved by employing monotonic optimization and semidefinite relaxation to find a performance upper bound. The problem obtained for TDMA is optimally solved by leveraging the time-selective nature of the IRS. Furthermore, for all three multiple access schemes, low-complexity suboptimal algorithms are developed by exploiting alternating optimization and successive convex approximation techniques, where a local region optimization method is applied for optimizing the IRS deployment location. Numerical results are provided to show that: 1) near-optimal performance can be achieved by the proposed suboptimal algorithms; 2) asymmetric and symmetric IRS deployment strategies are preferable for NOMA and FDMA/TDMA, respectively; 3) the performance gain achieved with IRS can be significantly improved by optimizing the deployment location.

Power Efficient IRS-Assisted NOMA

Abstract: In this paper, we propose a downlink multiple-input single-output (MISO) transmission scheme, which is assisted by an intelligent reflecting surface (IRS) consisting of a large number of passive reflecting elements. In the literature, it has been proved that nonorthogonal multiple access (NOMA) can achieve the same performance as computationally complex dirty paper coding, where the quasi-degradation condition is satisfied, conditioned on the users’ channels fall in the quasi-degradation region. However, in a conventional communication scenario, it is difficult to guarantee the quasi-degradation, because the channels are determined by the propagation environments and cannot be reconfigured. To overcome this difficulty, we focus on an IRS-assisted MISO NOMA system, where the wireless channels can be effectively tuned. We optimize the beamforming vectors and the IRS phase shift matrix for minimizing transmission power. Furthermore, we propose an improved quasi-degradation condition by using IRS, which can ensure that NOMA achieves the capacity region with high possibility. For a comparison, we study zero-forcing beamforming (ZFBF) as well, where the beamforming vectors and the IRS phase shift matrix are also jointly optimized. Comparing NOMA with ZFBF, it is shown that, with the same IRS phase shift matrix and the improved quasi-degradation condition, NOMA always outperforms ZFBF. At the same time, we identify the condition under which ZFBF outperforms NOMA, which motivates the proposed hybrid NOMA transmission. Simulation results show that the proposed IRS-assisted MISO system outperforms the MISO case without IRS, and the hybrid NOMA transmission scheme always achieves better performance than orthogonal multiple access.

MIMO-NOMA Networks Relying on Reconfigurable Intelligent Surface: A Signal Cancellation-Based Design

Abstract: Reconfigurable intelligent surface (RIS) technique stands as a promising signal enhancement or signal cancellation technique for next generation networks. We design a novel passive beamforming weight at RISs in a multiple-input multiple-output (MIMO) non-orthogonal multiple access (NOMA) network for simultaneously serving paired users, where a signal cancellation based (SCB) design is employed. In order to implement the proposed SCB design, we first evaluate the minimal required number of RISs in both the diffuse scattering and anomalous reflector scenarios. Then, new channel statistics are derived for characterizing the effective channel gains. In order to evaluate the network’s performance, we derive the closed-form expressions both for the outage probability (OP) and for the ergodic rate (ER). The diversity orders as well as the high-signal-to-noise-ratio (SNR) slopes are derived for engineering insights. Moreover, the network’s performance of a finite resolution design has been evaluated. Our analytical results demonstrate that: i) the inter-cluster interference can be eliminated with the aid of large number of RIS elements; ii) the line-of-sight of the BS-RIS and RIS-user links are required for the diffuse scattering scenario, whereas the LoS links are not required for the anomalous reflector scenario.

A Simple Design of IRS-NOMA Transmission

Abstract: This letter proposes a simple design of intelligent reflecting surface (IRS) assisted non-orthogonal multiple access (NOMA) downlink transmission. In particular, conventional spatial division multiple access (SDMA) is used first at the base station to generate orthogonal beams by using the spatial directions of the near users’ channels. Then, IRS-assisted NOMA is used to ensure that additional cell-edge users can also be served on these beams by aligning the cell-edge users’ effective channel vectors with the predetermined spatial directions. Both analytical and simulation results are provided to demonstrate the performance of the proposed IRS-NOMA scheme and also study the impact of hardware impairments on IRS-NOMA.

Intelligent Reflecting Surface Enhanced Multi-UAV NOMA Networks

Abstract: Intelligent reflecting surface (IRS) enhanced multi-unmanned aerial vehicle (UAV) non-orthogonal multiple access (NOMA) networks are investigated. A new transmission framework is proposed, where multiple UAV-mounted base stations employ NOMA to serve multiple groups of ground users with the aid of an IRS. The three-dimensional (3D) placement and transmit power of UAVs, the reflection matrix of the IRS, and the NOMA decoding orders among users are jointly optimized for maximization of the sum rate of considered networks. To tackle the formulated mixed-integer non-convex optimization problem with coupled variables, a block coordinate descent (BCD)-based iterative algorithm is developed. Specifically, the original problem is decomposed into three subproblems, which are alternately solved by exploiting the penalty-based method and the successive convex approximation technique. The proposed BCD-based algorithm is demonstrated to be able to obtain a stationary point of the original problem with polynomial time complexity. Numerical results show that: 1) the proposed NOMA-IRS scheme for multi-UAV networks achieves a higher sum rate compared to the benchmark schemes, i.e., orthogonal multiple access (OMA)-IRS and NOMA without IRS; 2) the use of IRS is capable of providing performance gain for multi-UAV networks by both enhancing channel qualities of UAVs to their served users and mitigating the inter-UAV interference; and 3) optimizing the UAV placement can make the sum rate gain brought by NOMA more distinct due to the flexible decoding order design.

Summary of existing research contributions on RIS-NOMA.

Reconfigurable Intelligent Surface (RIS)-NOMA

The aforementioned advantages of RIS-NOMA have drawn significant research interests in an effort to exploit the new DoFs introduced to NOMA by RISs. A summary of existing research contributions on RIS-NOMA is provided in Table {{table:6300e42f-be1b-4626-8a02-ab29bde34f0a}} . In particular, the authors of {{cite:37312cf}} considered an RIS-aided multi-user SISO BC. Considering the dynamic RIS configuration, the capacity and rate regions achieved by NOMA and OMA were characterized by jointly optimizing the power allocation of the AP and the phase shift of the RIS in each time slot. The derived solutions revealed that the optimal transmission strategy for OMA is alternating transmission between all users with their effective channel power gains maximized by the RIS. For NOMA, the optimal transmission strategy is alternating transmission between different user groups with dynamic decoding orders facilitated by the RIS. Moreover, the results in {{cite:37312cf}} also showed that the capacity gain of NOMA is less sensitive to the dynamic RIS configuration than the rate gain of OMA. This revealed that, compared to OMA, NOMA not only achieves a higher capacity gain but also requires less hardware complexity at the RIS, because the dynamic RIS configuration needed for OMA requires considerable overhead for information exchange between the AP and the RIS. The capacity/rate region characterization problem of the RIS-aided two-user SISO MAC and BC with NOMA and OMA was further investigated in {{cite:e2389ae}} by leveraging the rate-profile method and uplink-downlink duality, where two types of network-level RIS deployment strategies were considered, namely centralized deployment and distributed deployment. In particular, one large-size RIS was deployed near the AP for the centralized deployment strategy, while two small-size RISs were separately deployed near each user for the distributed deployment strategy. In both strategies, the total number of RIS elements was assumed to be the same for a fair comparison. One interesting insight obtained in {{cite:e2389ae}} is that regardless of the employed multiple access scheme, the centralized deployment strategy always outperforms the distributed one in terms of the capacity/rate region. As a further advancement, the authors of {{cite:113c618}} investigated the link-level optimal RIS deployment design for an RIS-aided multi-user downlink network. Considering NOMA, TDMA, and FDMA transmission, a weighted sum rate maximization problem was formulated by jointly optimizing the power allocation of the AP and the phase shifts and deployment location of the RIS. Both monotonic optimization and alternating optimization based algorithms were developed to find a performance upper bound and a suboptimal solution, respectively, which were shown to achieve a similar performance via simulations. One useful guideline for RIS deployment revealed in {{cite:113c618}} was that an asymmetric RIS deployment strategy is preferable for NOMA, while a symmetric RIS deployment strategy is superior for OMA. In Fig. {{figure:68969666-5bf0-48cd-9270-a5c74a0be45d}} , we present the obtained maximum weighted sum rate obtained by different multiple access schemes in {{cite:113c618}}. As can be observed, NOMA significantly outperforms both TDMA and FDMA. The performance gain is more pronounced as the number of RIS elements increases, which confirms the advantages of the integration of NOMA and RISs.


Modeling the Intensity Function of Point Process Via Recurrent Neural Networks

Abstract: Event sequence, asynchronously generated with random timestamp, is ubiquitous among applications. The precise and arbitrary timestamp can carry important clues about the underlying dynamics, and has lent the event data fundamentally different from the time-series whereby series is indexed with fixed and equal time interval. One expressive mathematical tool for modeling event is point process. The intensity functions of many point processes involve two components: the background and the effect by the history. Due to its inherent spontaneousness, the background can be treated as a time series while the other need to handle the history events. In this paper, we model the background by a Recurrent Neural Network (RNN) with its units aligned with time series indexes while the history effect is modeled by another RNN whose units are aligned with asynchronous events to capture the long-range dynamics. The whole model with event type and timestamp prediction output layers can be trained end-to-end. Our approach takes an RNN perspective to point process, and models its background and history effect. For utility, our method allows a black-box treatment for modeling the intensity which is often a pre-defined parametric form in point processes. Meanwhile end-to-end training opens the venue for reusing existing rich techniques in deep network for point process modeling. We apply our model to the predictive maintenance problem using a log dataset by more than 1000 ATMs from a global bank headquartered in North America.

Fully Neural Network based Model for General Temporal Point Processes

Abstract: A temporal point process is a mathematical model for a time series of discrete events, which covers various applications. Recently, recurrent neural network (RNN) based models have been developed for point processes and have been found effective. RNN based models usually assume a specific functional form for the time course of the intensity function of a point process (e.g., exponentially decreasing or increasing with the time since the most recent event). However, such an assumption can restrict the expressive power of the model. We herein propose a novel RNN based model in which the time course of the intensity function is represented in a general manner. In our approach, we first model the integral of the intensity function using a feedforward neural network and then obtain the intensity function as its derivative. This approach enables us to both obtain a flexible model of the intensity function and exactly evaluate the log-likelihood function, which contains the integral of the intensity function, without any numerical approximations. Our model achieves competitive or superior performances compared to the previous state-of-the-art methods for both synthetic and real datasets.

Transformer Hawkes Process

Abstract: Modern data acquisition routinely produce massive amounts of event sequence data in various domains, such as social media, healthcare, and financial markets. These data often exhibit complicated short-term and long-term temporal dependencies. However, most of the existing recurrent neural network based point process models fail to capture such dependencies, and yield unreliable prediction performance. To address this issue, we propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies and meanwhile enjoys computational efficiency. Numerical experiments on various datasets show that THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. Moreover, THP is quite general and can incorporate additional structural knowledge. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.

Summary of some neural marked point processes.

Existing neural marked point processes can be categorized into three types in terms of the network architecture: recurrent, attentive and graph-based, as shown in Figure {{figure:78413454-1b69-45db-8136-84b20f7c40b9}} . In Table {{table:c1c4d093-aadc-45ba-adc8-967fc92efbbc}} , we summarize some important models in terms of the network architecture and the intensity function.


Robust Functional Pruning Optimal
Partitioning

Bayesian Online Changepoint Detection

Abstract: Changepoints are abrupt variations in the generative parameters of a data sequence. Online detection of changepoints is useful in modelling and prediction of time series in application areas such as finance, biometrics, and robotics. While frequentist methods have yielded online filtering and prediction techniques, most Bayesian papers have focused on the retrospective segmentation problem. Here we examine the case where the model parameters before and after the changepoint are independent and we derive an online algorithm for exact inference of the most recent changepoint. We compute the probability distribution of the length of the current ``run,'' or time since the last changepoint, using a simple message-passing algorithm. Our implementation is highly modular so that the algorithm may be applied to a variety of types of data. We illustrate this modularity by demonstrating the algorithm on three different real-world data sets.

Spatio-temporal Bayesian On-line Changepoint Detection with Model Selection

Abstract: Bayesian On-line Changepoint Detection is extended to on-line model selection and non-stationary spatio-temporal processes. We propose spatially structured Vector Autoregressions (VARs) for modelling the process between changepoints (CPs) and give an upper bound on the approximation error of such models. The resulting algorithm performs prediction, model selection and CP detection on-line. Its time complexity is linear and its space complexity constant, and thus it is two orders of magnitudes faster than its closest competitor. In addition, it outperforms the state of the art for multivariate data.

A computationally efficient nonparametric approach for changepoint detection

Abstract: In this paper we build on an approach proposed by Zou et al. (2014) for nonpara- metric changepoint detection. This approach defines the best segmentation for a data set as the one which minimises a penalised cost function, with the cost function defined in term of minus a non-parametric log-likelihood for data within each segment. Min- imising this cost function is possible using dynamic programming, but their algorithm had a computational cost that is cubic in the length of the data set. To speed up computation, Zou et al. (2014) resorted to a screening procedure which means that the estimated segmentation is no longer guaranteed to be the global minimum of the cost function. We show that the screening procedure adversely affects the accuracy of the changepoint detection method, and show how a faster dynamic programming algorithm, Pruned Exact Linear Time, PELT (Killick et al., 2012), can be used to find the optimal segmentation with a computational cost that can be close to linear in the amount of data. PELT requires a penalty to avoid under/over-fitting the model which can have a detrimental effect on the quality of the detected changepoints. To overcome this issue we use a relatively new method, Changepoints Over a Range of PenaltieS (CROPS) (Haynes et al., 2015), which finds all of the optimal segmentations for multiple penalty values over a continuous range. We apply our method to detect changes in heart rate during physical activity.

A Nonparametric Approach for Multiple Change Point Analysis of Multivariate Data

Abstract: Change point analysis has applications in a wide variety of fields. The general problem concerns the inference of a change in distribution for a set of time-ordered observations. Sequential detection is an online version in which new data are continually arriving and are analyzed adaptively. We are concerned with the related, but distinct, offline version, in which retrospective analysis of an entire sequence is performed. For a set of multivariate observations of arbitrary dimension, we consider nonparametric estimation of both the number of change points and the positions at which they occur. We do not make any assumptions regarding the nature of the change in distribution or any distribution assumptions beyond the existence of the αth absolute moment, for some α ∈ (0, 2). Estimation is based on hierarchical clustering and we propose both divisive and agglomerative algorithms. The divisive method is shown to provide consistent estimates of both the number and the location of change points under standard regularity assumptions. We compare the proposed approach with competing methods in a simulation study. Methods from cluster analysis are applied to assess performance and to allow simple comparisons of location estimates, even when the estimated number differs. We conclude with applications in genetics, finance, and spatio-temporal analysis. Supplementary materials for this article are available online.

Optimal detection of changepoints with a linear computational cost

Abstract: We consider the problem of detecting multiple changepoints in large data sets. Our focus is on applications where the number of changepoints will increase as we collect more data: for example in genetics as we analyse larger regions of the genome, or in finance as we observe time-series over longer periods. We consider the common approach of detecting changepoints through minimising a cost function over possible numbers and locations of changepoints. This includes several established procedures for detecting changing points, such as penalised likelihood and minimum description length. We introduce a new ∗R. Killick is Senior Research Associate, Department of Mathematics & Statistics, Lancaster University, Lancaster, UK (E-mail: r.killick@lancs.ac.uk). P. Fearnhead is Professor, Department of Mathematics & Statistics, Lancaster University, Lancaster, UK (E-mail: p.fearnhead@lancs.ac.uk). I.A. Eckley is Senior Lecturer, Department of Mathematics & Statistics, Lancaster University, Lancaster, UK (E-mail: i.eckley@lancs.ac.uk). The authors are grateful to Richard Davis and Alice Cleynen for providing the Auto-PARM and PDPA software respectively. Part of this research was conducted whilst R. Killick was a jointly funded Engineering and Physical Sciences Research Council (EPSRC) / Shell Research Ltd graduate student at Lancaster University. Both I.A. Eckley and R. Killick also gratefully acknowledge the financial support of the EPSRC grant number EP/I016368/1. 1 ar X iv :1 10 1. 14 38 v3 [ st at .M E ] 9 O ct 2 01 2 method for finding the minimum of such cost functions and hence the optimal number and location of changepoints that has a computational cost which, under mild conditions, is linear in the number of observations. This compares favourably with existing methods for the same problem whose computational cost can be quadratic or even cubic. In simulation studies we show that our new method can be orders of magnitude faster than these alternative exact methods. We also compare with the Binary Segmentation algorithm for identifying changepoints, showing that the exactness of our approach can lead to substantial improvements in the accuracy of the inferred segmentation of the data.

Changepoint Detection in the Presence of Outliers

Abstract: Many traditional methods for identifying changepoints can struggle in the presence of outliers, or when the noise is heavy-tailed. Often they will infer additional changepoints in order to fit the outliers. To overcome this problem, data often needs to be pre-processed to remove outliers, though this is difficult for applications where the data needs to be analysed online. We present an approach to changepoint detection that is robust to the presence of outliers. The idea is to adapt existing penalised cost approaches for detecting changes so that they use loss functions that are less sensitive to outliers. We argue that loss functions that are bounded, such as the classical biweight loss, are particularly suitable -- as we show that only bounded loss functions are robust to arbitrarily extreme outliers. We present an efficient dynamic programming algorithm that can find the optimal segmentation under our penalised cost criteria. Importantly, this algorithm can be used in settings where the data needs to be analysed online. We show that we can consistently estimate the number of changepoints, and accurately estimate their locations, using the biweight loss function. We demonstrate the usefulness of our approach for applications such as analysing well-log data, detecting copy number variation, and detecting tampering of wireless devices.

Wild binary segmentation for multiple change-point detection

Abstract: We propose a new technique, called wild binary segmentation (WBS), for consistent estimation of the number and locations of multiple change-points in data. We assume that the number of change-points can increase to infinity with the sample size. Due to a certain random localisation mechanism, WBS works even for very short spacings between the change-points and/or very small jump magnitudes, unlike standard binary segmentation. On the other hand, despite its use of localisation, WBS does not require the choice of a window or span parameter, and does not lead to a significant increase in computational complexity. WBS is also easy to code. We propose two stopping criteria for WBS: one based on thresholding and the other based on what we term the `strengthened Schwarz information criterion'. We provide default recommended values of the parameters of the procedure and show that it offers very good practical performance in comparison with the state of the art. The WBS methodology is implemented in the R package wbs, available on CRAN. In addition, we provide a new proof of consistency of binary segmentation with improved rates of convergence, as well as a corresponding result for WBS.

An Evaluation of Change Point Detection Algorithms

Abstract: Change point detection is an important part of time series analysis, as the presence of a change point indicates an abrupt and significant change in the data generating process. While many algorithms for change point detection have been proposed, comparatively little attention has been paid to evaluating their performance on real-world time series. Algorithms are typically evaluated on simulated data and a small number of commonly-used series with unreliable ground truth. Clearly this does not provide sufficient insight into the comparative performance of these algorithms. Therefore, instead of developing yet another change point detection method, we consider it vastly more important to properly evaluate existing algorithms on real-world data. To achieve this, we present a data set specifically designed for the evaluation of change point detection algorithms that consists of 37 time series from various application domains. Each series was annotated by five human annotators to provide ground truth on the presence and location of change points. We analyze the consistency of the human annotators, and describe evaluation metrics that can be used to measure algorithm performance in the presence of multiple ground truth annotations. Next, we present a benchmark study where 14 algorithms are evaluated on each of the time series in the data set. Our aim is that this data set will serve as a proving ground in the development of novel change point detection algorithms.

The change point detection algorithms considered in thebenchmark study.

The aim of the study is to obtain an understanding of the performance of
existing change point methods on real-world data. With that in mind we
evaluate a large selection of methods that are either frequently used or that
have been recently developed. Table {{table:14f6d93a-4d71-438b-9705-93514627bc75}}  lists the methods
included in the study. To avoid issues with implementations we use existing
software packages for all methods, which are almost always made available by
the authors themselves. We also include the baseline zero method,
which always returns that a series contains no change points.


Scalability in Perception for Autonomous Driving: Waymo Open Dataset

Abstract: The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the over-all viability of the technology. In an effort to help align the research community’s contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.

nuScenes: A Multimodal Dataset for Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.

Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations

Abstract: 3D object detection has recently become popular due to many applications in robotics, augmented reality, autonomy, and image retrieval. We introduce the Objectron dataset to advance the state of the art in 3D object detection and foster new research and applications, such as 3D object tracking, view synthesis, and improved 3D shape representation. The dataset contains object-centric short videos with pose annotations for nine categories and includes 4 million annotated images in 14, 819 annotated videos. We also propose a new evaluation metric, 3D Intersection over Union, for 3D object detection. We demonstrate the usefulness of our dataset in 3D object detection and novel view synthesis tasks by providing baseline models trained on this dataset. Our dataset and evaluation source code are available online at Github.com/google-research-datasets/Objectron.

Overview of the described datasets

Similar to the domain focus, the most commonly used datasets for 3DOD developments can be roughly divided into two groups, distinguishing between autonomous driving scenarios and indoor scenes.
In the following, we describe some of the major datasets that are publicly available. In addition, Table {{table:975315bf-5f8d-4af1-95f1-8e4d96febe26}}  provides an overview of the described datasets including the main reference and the environment of the recordings as well as the number of scenes, frames, 3D bounding boxes and object classes.


Fine-grained classification of aircrafts

Land use & cover classification with satellite images

Natural adversarial examples of 200 ImageNet classes

“a photo of a [class], a type of aircraft.”

“a centered satellite photo of [class].”

Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Abstract: We introduce UCF101 which is currently the largest dataset of human actions. It consists of 101 action classes, over 13k clips and 27 hours of video data. The database consists of realistic user uploaded videos containing camera motion and cluttered background. Additionally, we provide baseline action recognition results on this new dataset using standard bag of words approach with overall performance of 44.5%. To the best of our knowledge, UCF101 is currently the most challenging dataset of actions due to its large number of classes, large number of clips and also unconstrained nature of such clips.

Natural Adversarial Examples

Abstract: We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets’ real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called IMAGENET-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called IMAGENET-O, which is the first out-of-distribution detection dataset created for ImageNet models. On IMAGENET-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on IMAGENET-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

Abstract: We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pre-trained with 1000× more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness.

Summary of 11 datasets for few-shot learning and 4 target datasets of domain generalization. The 7 selected templates {{cite:9b28d52}} for ImageNet series datasets are “itap of a [class].”, “a bad photo of the [class].”, “a origami [class].”, “a photo of the large [class].”, “a [class] in a video game.”, “art of the [class].” and “a photo of the small [class].”.

In our main text, we evaluate the proposed method for few-shot learning tasks on 11 benchmark datasets and domain generalization tasks on ImageNet to its variants (ImageNet-V2, -Sketch, -A and -R). We summarize the dataset information in Table {{table:2f3a1921-82fa-4275-a388-e9ef43d06db7}} .

For the generalization datasets, ImageNet-V2 and ImageNet-Sketch share the same label space with ImageNet (1000 classes), while the label spaces of ImageNet-A (200 classes) and ImageNet-R (200 classes) are both sub-spaces of the ImageNet label space.
The variants of ImageNet contain substantially different data distributions (See Table {{table:2f3a1921-82fa-4275-a388-e9ef43d06db7}}  Description) from ImageNet, which makes them satisfactory domain generalization benchmarks.
Following CoOp {{cite:d6235cb}}, we choose ImageNet as the source domain data while the variants as the target one.


Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems

Abstract: Solving math word problems is a challenging task that requires accurate natural language understanding to bridge natural language texts and math expressions. Motivated by the intuition about how human generates the equations given the problem texts, this paper presents a neural approach to automatically solve math word problems by operating symbols according to their semantic meanings in texts. This paper views the process of generating equation as a bridge between the semantic world and the symbolic world, where the proposed neural math solver is based on an encoder-decoder framework. In the proposed model, the encoder is designed to understand the semantics of problems, and the decoder focuses on tracking semantic meanings of the generated symbols and then deciding which symbol to generate next. The preliminary experiments are conducted in a dataset Math23K, and our model significantly outperforms both the state-of-the-art single model and the best non-retrieval-based model over about 10% accuracy, demonstrating the effectiveness of bridging the symbolic and semantic worlds from math word problems.

Neural Machine Translation by Jointly Learning to Align and Translate

Abstract: Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.

Variational Neural Machine Translation

Abstract: Models of neural machine translation are often from a discriminative family of encoderdecoders that learn a conditional distribution of a target sentence given a source sentence. In this paper, we propose a variational model to learn this conditional distribution for neural machine translation: a variational encoderdecoder model that can be trained end-to-end. Different from the vanilla encoder-decoder model that generates target translations from hidden representations of source sentences alone, the variational model introduces a continuous latent variable to explicitly model underlying semantics of source sentences and to guide the generation of target translations. In order to perform efficient posterior inference and large-scale training, we build a neural posterior approximator conditioned on both the source and the target sides, and equip it with a reparameterization technique to estimate the variational lower bound. Experiments on both Chinese-English and English- German translation tasks show that the proposed variational neural machine translation achieves significant improvements over the vanilla neural machine translation baselines.

Attention Is All You Need

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Semantically-Aligned Universal Tree-Structured Solver for Math Word Problems

Abstract: A practical automatic textual math word problems (MWPs) solver should be able to solve various textual MWPs while most existing works only focused on one-unknown linear MWPs. Herein, we propose a simple but efficient method called Universal Expression Tree (UET) to make the first attempt to represent the equations of various MWPs uniformly. Then a semantically-aligned universal tree-structured solver (SAU-Solver) based on an encoder-decoder framework is proposed to resolve multiple types of MWPs in a unified model, benefiting from our UET representation. Our SAU-Solver generates a universal expression tree explicitly by deciding which symbol to generate according to the generated symbols' semantic meanings like human solving MWPs. Besides, our SAU-Solver also includes a novel subtree-level semanticallyaligned regularization to further enforce the semantic constraints and rationality of the generated expression tree by aligning with the contextual information. Finally, to validate the universality of our solver and extend the research boundary of MWPs, we introduce a new challenging Hybrid Math Word Problems dataset (HMWP), consisting of three types of MWPs. Experimental results on several MWPs datasets show that our model can solve universal types of MWPs and outperforms several state-of-the-art models.

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

The implemented models in MWPToolkit. Currently, the toolkit includes four types of models: Seq2Seq, Seq2Tree, Graph2Tree, and Pretrained models.

We have carefully surveyed the recent literatures and selected the commonly-used MWP solving models in our library.
As the first released version, we have implemented 17 MWP solving models in the four categories: Seq2seq, Seq2tree, Graph2tree, and Pretrained Language Models.
In the future, more methods will be added into our toolkit as the regular update, like EPT {{cite:e23a0f7}}, KAS2T {{cite:aecf29b}}, and NumS2T {{cite:0431735}}.
We summarize all the implemented models in Table {{table:f047bcc8-e3ee-4cc0-8deb-62ba982ed6c1}} .


One desired user, multiple eavesdroppers, i.i.d. Rician

One desired user, one eavesdropper, perfect CSI

Relay-aided, one desired user, one eavesdropper, i.i.d. Rayleigh

Secure transmission with finite alphabet inputs

Secrecy performance analysis and power allocation designs

Jamming-Aided Secure Communication in Massive MIMO Rician Channels

Abstract: In this paper, we investigate the artificial noise-aided jamming design for a transmitter equipped with large antenna array in Rician fading channels. We figure out that when the number of transmit antennas tends to infinity, whether the secrecy outage happens in a Rician channel depends on the geometric locations of eavesdroppers. In this light, we first define and analytically describe the secrecy outage region (SOR), indicating all possible locations of an eavesdropper that can cause secrecy outage. After that, the secrecy outage probability (SOP) is derived, and a jamming-beneficial range, i.e., the distance range of eavesdroppers which enables uniform jamming to reduce the SOP, is determined. Then, the optimal power allocation between messages and artificial noise is investigated for different scenarios. Furthermore, to use the jamming power more efficiently and further reduce the SOP, we propose directional jamming that generates jamming signals at selected beams (mapped to physical angles) only, and power allocation algorithms are proposed for the cases with and without the information of the suspicious area, i.e., possible locations of eavesdroppers. We further extend the discussions to multiuser and multi-cell scenarios. At last, numerical results validate our conclusions and show the effectiveness of our proposed jamming power allocation schemes.

Secure Transmission With Large Numbers of Antennas and Finite Alphabet Inputs

Abstract: In this paper, we investigate secure transmission over the large-scale multiple-antenna wiretap channel with finite alphabet inputs. First, we investigate the case where instantaneous channel state information (CSI) of the eavesdropper is known at the transmitter. We show analytically that a generalized singular value decomposition (GSVD)-based design, which is optimal for Gaussian inputs, may exhibit a severe performance loss for finite alphabet inputs in the high signal-to-noise ratio regime. In light of this, we propose a novel Per-Group-GSVD (PG-GSVD) design, which can effectively compensate the performance loss caused by the GSVD design. More importantly, the computational complexity of the PG-GSVD design is by orders of magnitude lower than that of the existing design for finite alphabet inputs while the resulting performance loss is minimal. Then, we extend the PG-GSVD design to the case where only statistical CSI of the eavesdropper is available at the transmitter. Numerical results indicate that the proposed PG-GSVD design can be efficiently implemented in large-scale multiple-antenna systems and achieves significant performance gains compared with the GSVD design.

Large-Scale MIMO Relaying Techniques for Physical Layer Security: AF or DF?

Abstract: In this paper, we consider a large scale multiple input multiple output (LS-MIMO) relaying system, where an information source sends the message to its intended destination aided by an LS-MIMO relay, while a passive eavesdropper tries to intercept the information forwarded by the relay. The advantage of a large scale antenna array is exploited to improve spectral efficiency and enhance wireless security. In particular, the challenging issue incurred by short-distance interception is well addressed. Under very practical assumptions, i.e., no eavesdropper channel state information (CSI) and imperfect legitimate CSI at the relay, this paper gives a thorough secrecy performance analysis and comparison of two classic relaying techniques, i.e., amplify-and-forward (AF) and decode-and-forward (DF). Furthermore, asymptotical analysis is carried out to provide clear insights on the secrecy performance for such an LS-MIMO relaying system. We show that under large transmit powers, AF is a better choice than DF from the perspectives of both secrecy performance and implementation complexity, and prove that there exits an optimal transmit power at medium regime that maximizes the secrecy outage capacity.

Secure Massive MIMO with Passive Eavesdropper

Physical layer security for massive MIMO systems with
passive eavesdroppers has been recently studied.
J. Zhu et al. study secure massive MIMO transmissions for multi-cell multi-user systems over
i.i.d. Rayleigh fading channel {{cite:4e3a586}}, where a passive eavesdropper attempts to decode the information sent to one of the users.
The impact of multi-cell interference and pilot contamination on the achievable erogdic secrecy rate
are analyzed and several matched filtering precoding and artificial noise (AN) generation designs are proposed
to degrade the eavesdropper's channel and protect the desired user's channel.
For the same system model, regularized channel inversion and AN transmission schemes
are designed in {{cite:1c886a2}} to further improve the secrecy rate performance.
J. Wang et al. investigate AN-aided secure massive MIMO transmission
over i.i.d. Rician fading channel {{cite:bd7f3fd}}.
For single-cell multiuser massive MIMO systems with
distributed antennas, K. Guo et al. design
three secure-constrained power allocation schemes {{cite:7bf76e8}} by maximizing the minimum user's signal-to-interference-noise ratio (SINR)
subject to the eavesdropper's SINR and the sum power constraint and minimizing the sum transmit power
subject to SINR constraints of users and the eavesdropper, respectively.
Y. Wu et al. investigate secure transmission designs for large-scale MIMO systems
with finite alphabet inputs {{cite:d8181db}}. Power allocation schemes for relay-aided large-scale
MIMO systems are proposed in {{cite:8c23fa9}}, {{cite:68dcfd2}}.
A brief summary of above work is given in Table {{table:4532fc69-89dd-47c3-a009-28da15005139}} .


AST: Audio Spectrogram Transformer

Abstract: In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.

Efficient Training of Audio Transformers with Patchout

Abstract: The great success of transformer-based models in natural language processing (NLP) has led to various attempts at adapting these architectures to other domains such as vision and audio. Recent work has shown that transformers can outperform Convolutional Neural Networks (CNNs) on vision and audio tasks. However, one of the main shortcomings of transformer models, compared to the well-established CNNs, is the computational complexity. In transformers, the compute and memory complexity is known to grow quadratically with the input length. Therefore, there has been extensive work on optimizing transformers, but often at the cost of degrading predictive performance. In this work, we propose a novel method to optimize and regularize transformers on audio spectrograms. Our proposed models achieve a new state-of-the-art performance on Audioset and can be trained on a single consumer-grade GPU. Furthermore, we propose a transformer model that outperforms CNNs in terms of both performance and training speed. Source code: https://github.com/kkoutini/PaSST

Masked Spectrogram Prediction for Self-Supervised Audio Pre-Training

Abstract: Transformer-based models attain excellent results and generalize well when trained on sufficient amounts of data. However, constrained by the limited data available in the audio domain, most transformer-based models for audio tasks are finetuned from pre-trained models in other domains (e.g. image), which has a notable gap with the audio domain. Other methods explore the self-supervised learning approaches directly in the audio domain but currently do not perform well in the downstream tasks. In this paper, we present a novel self-supervised learning method for transformer-based audio models, called masked spectrogram prediction (MaskSpec), to learn powerful audio representations from unlabeled audio data (AudioSet used in this paper). Our method masks random patches of the input spectrogram and reconstructs the masked regions with an encoder-decoder architecture. Experimental results demonstrate MaskSpec reaches the performance of 0.471 (mAP) on AudioSet, 0.854 (mAP) on Open-MIC2018, 0.982 (accuracy) on ESC-50, 0.976 (accuracy) on SCV2, and 0.823 (accuracy) on DCASE2019 Task1A. The source code and pre-trained models have been released.1

BEATs: Audio Pre-Training with Acoustic Tokenizers

Abstract: The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.

Comparison transformers from the literature in terms of initialization weights, number of GPUs used for training, training time, and mAP obtained in AudioSet.

Transformers in audio classification tasks

Transformers have become a popular choice for audio tasks due to their superior performance compared to their convolutional counterparts when sufficient data is available.
Lately, AudioSet, with almost 2 M audio event excerpts, has become a popular benchmark led by transformer models.
A popular approach consists of applying attention over small overlapping patches (e.g., 16 {{formula:d9ba804b-0a13-4022-8f47-2964dc98ecd2}}  16) from the spectrogram using a classification objective.
The sequence of spectrogram patches is linearly projected to a 1-D space where a trainable positional encoding signal is added.
A trainable classification token is appended to the sequence of projections, and after a number of Transformer blocks it is used to solve the classification task using a linear classifier.
This idea was first introduced in the image domain by ViT {{cite:4fe4176}} and adapted to audio spectrograms in AST {{cite:d561092}}.
PaSST extends this approach by introducing patchout, a technique consisting of discarding random patches from the input spectrogram at training time (see Figure {{figure:5ae43a38-e292-4d19-bb13-1d1bbe3bcdba}} ) {{cite:58ba2a5}}.
This technique has two benefits.
First, by discarding input patches, the training sequence length is significantly reduced, which increases the training speed.
Second, it acts as a regularization technique that improves the robustness of the transformer.
Additionally, patchout can be combined with other training methods.
MaskSpec is a self-supervised pre-training method based on an encoder-decoder architecture where the decoder has to reconstruct the spectrogram from a partial spectrogram altered with patchout {{cite:8335422}}.
Beats is a transformer trained with a supervised objective and patchout where the labels come from a codebook of randomly initialized vectors that is iteratively optimized {{cite:c2b51fd}}.
While these techniques prevent the transformers from depending on initializing from weights of pre-trained models, such systems are significantly more resource-demanding.
Table {{table:267793dc-a174-4431-bb1d-984d53d1825c}}  compares the mentioned audio transformers in terms of GPUs used for training, training duration, and mean Average Precision (mAP) on AudioSet.
Remarkably, PaSST achieves an excellent trade-off between mAP and needed resources.
Since we aim to use transformer models that can be trained with a computational budget equivalent to SOTA CNNs (i.e., using consumer-grade GPUs), we focus on the standard patchout training with a supervised objective.


Total CT slices: 270
COVID: 120
Normal: 150

Encoding and decoding transformations performed are also recommended for medical imaging segmentation.

Minimize the number of parameters and less computational time to provide accuracy.

They have segmented the infected regions of the lungs in a loop method.

Coronavirus Detection and Analysis on Chest CT with Deep Learning

Abstract: The outbreak of the novel coronavirus, officially declared a global pandemic, has a severe impact on our daily lives. As of this writing there are approximately 197,188 confirmed cases of which 80,881 are in "Mainland China" with 7,949 deaths, a mortality rate of 3.4%. In order to support radiologists in this overwhelming challenge, we develop a deep learning based algorithm that can detect, localize and quantify severity of COVID-19 manifestation from chest CT scans. The algorithm is comprised of a pipeline of image processing algorithms which includes lung segmentation, 2D slice classification and fine grain localization. In order to further understand the manifestations of the disease, we perform unsupervised clustering of abnormal slices. We present our results on a dataset comprised of 110 confirmed COVID-19 patients from Zhejiang province, China.

Fully convolutional networks for semantic segmentation

Abstract: Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet [20], the VGG net [31], and GoogLeNet [32]) into fully convolutional networks and transfer their learned representations by fine-tuning [3] to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes less than one fifth of a second for a typical image.

Lung Infection Quantification of COVID-19 in CT Images with Deep Learning

Abstract: CT imaging is crucial for diagnosis, assessment and staging COVID-19 infection. Follow-up scans every 3-5 days are often recommended for disease progression. It has been reported that bilateral and peripheral ground glass opacification (GGO) with or without consolidation are predominant CT findings in COVID-19 patients. However, due to lack of computerized quantification tools, only qualitative impression and rough description of infected areas are currently used in radiological reports. In this paper, a deep learning (DL)-based segmentation system is developed to automatically quantify infection regions of interest (ROIs) and their volumetric ratios w.r.t. the lung. The performance of the system was evaluated by comparing the automatically segmented infection regions with the manually-delineated ones on 300 chest CT scans of 300 COVID-19 patients. For fast manual delineation of training samples and possible manual intervention of automatic results, a human-in-the-loop (HITL) strategy has been adopted to assist radiologists for infection region segmentation, which dramatically reduced the total segmentation time to 4 minutes after 3 iterations of model updating. The average Dice simiarility coefficient showed 91.6% agreement between automatic and manual infaction segmentations, and the mean estimation error of percentage of infection (POI) was 0.3% for the whole lung. Finally, possible applications, including but not limited to analysis of follow-up CT scans and infection distributions in the lobes and segments correlated with clinical findings, were discussed.

Comparison of Segmentation models for COVID identification

Segmentation models can be used to detect the infected lung region. Mainly identify the shape of the present part. It specifies the defect inside the COVID-19 chest CT images. Some of the sample models like U-Net {{cite:c243669}}, FCN {{cite:80cbdcd}}, VB-Net {{cite:016de6c}} are discussed in Table {{table:b892ab90-1220-45ea-99d3-d900ea1d2da5}} . These models are specified accurately in the symptomatic region in the lungs. Using segmentation algorithms closed with related region merging techniques to perform specific actions. Segmentation performed in the particular areas of the lungs that predicted areas are changed into pixel predictions. Pixel can be labeled depending on the scoring system. Segmentation models handle some other diseases presented inside the lungs. These models characterize the number of CT images for the training and testing process. A fully Convolutional Network (FCN) has connected layers as sampling, pooling, and convolution. U-Net model based on decoder-encoder architecture, fast and exact segmentation of images.


The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

Abstract: We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 (Chung et al., 2022). Through careful ablation studies on the Flan Collection of tasks and methods, we tease apart the effect of design decisions which enable Flan-T5 to outperform prior work by 3-17%+ across evaluation settings. We find task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings (zero-shot, few-shot, and chain-of-thought) actually yields stronger (2%+) performance in all settings. In further experiments, we show Flan-T5 requires less finetuning to converge higher and faster than T5 on single downstream tasks, motivating instruction-tuned models as more computationally-efficient starting checkpoints for new tasks. Finally, to accelerate research on instruction tuning, we make the Flan 2022 collection of datasets, templates, and methods publicly available at https://github.com/google-research/FLAN/tree/main/flan/v2.

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Abstract: Large “instruction-tuned” language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning.

Cross-Task Generalization via Natural Language Crowdsourcing Instructions

Abstract: Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.

List of open-source instruction-tuning datasets.

Arguably, the core of instruction tuning is the instruction data that are used to train foundation LLMs.
For instance, the quality, quantity, diversity, and format can all determine the behavior of the instructed model.
Hence, we collect details of several open-source instruction datasets in Table {{table:c43749b6-b90e-460d-8f11-4ad45ee15ecc}} .
Notably, we have observed a growing trend of leveraging synthetic instruction data from closed-source models.
While this practice may allow instructed models to mimic the behavior of models such as GPT-4, this may lead to issues such as inheriting the black-box nature of closed-source models, and instability due to noisy synthetic instructions.


Context Encoding for Semantic Segmentation

Abstract: Recent work has made significant progress in improving spatial resolution for pixelwise labeling with Fully Convolutional Network (FCN) framework by employing Dilated/Atrous convolution, utilizing multi-scale features and refining boundaries. In this paper, we explore the impact of global contextual information in semantic segmentation by introducing the Context Encoding Module, which captures the semantic context of scenes and selectively highlights class-dependent featuremaps. The proposed Context Encoding Module significantly improves semantic segmentation results with only marginal extra computation cost over FCN. Our approach has achieved new state-of-the-art results 51.7% mIoU on PASCAL-Context, 85.9% mIoU on PASCAL VOC 2012. Our single model achieves a final score of 0.5567 on ADE20K test set, which surpasses the winning entry of COCO-Place Challenge 2017. In addition, we also explore how the Context Encoding Module can improve the feature representation of relatively shallow networks for the image classification on CIFAR-10 dataset. Our 14 layer network has achieved an error rate of 3.45%, which is comparable with state-of-the-art approaches with over 10Ã— more layers. The source code for the complete system are publicly available1.

Rethinking Atrous Convolution for Semantic Image Segmentation

Abstract: In this work, we revisit atrous convolution, a powerful tool to explicitly adjust filter's field-of-view as well as control the resolution of feature responses computed by Deep Convolutional Neural Networks, in the application of semantic image segmentation. To handle the problem of segmenting objects at multiple scales, we design modules which employ atrous convolution in cascade or in parallel to capture multi-scale context by adopting multiple atrous rates. Furthermore, we propose to augment our previously proposed Atrous Spatial Pyramid Pooling module, which probes convolutional features at multiple scales, with image-level features encoding global context and further boost performance. We also elaborate on implementation details and share our experience on training our system. The proposed `DeepLabv3' system significantly improves over our previous DeepLab versions without DenseCRF post-processing and attains comparable performance with other state-of-art models on the PASCAL VOC 2012 semantic image segmentation benchmark.

Pyramid Scene Parsing Network

Abstract: Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed pyramid scene parsing network (PSPNet). Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction. The proposed approach achieves state-of-the-art performance on various datasets. It came first in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark and Cityscapes benchmark. A single PSPNet yields the new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes.

Pyramid Scene Parsing Network

Performance on the val set of Pascal Context dataset with the ResNet-50 as the backbone.

To show the effectiveness of the proposed method, we conduct a systematical ablation study on Pascal Context dataset with ResNet-50 as the backbone, as shown in Table {{table:eae837dc-ddfd-4800-a13f-30af82c4ce37}} .
We report the standard evaluation metrics of pixel accuracy (pixAcc) and mean Intersection of Union (mIoU).
Notably, no multi-scale testing and left-right flipping are applied to the val images.

For methods in DilatedFCN, the downsampling operations in the last two convolution stages are removed, resulting in the output stride (OS) to be 8.
Encoding-8-None in Table {{table:eae837dc-ddfd-4800-a13f-30af82c4ce37}}  represents the original EncNet {{cite:2862f5b}}.
To show the effect of dilated convolutions, we replace the backbone of EncNet with that of the original FCN (the same as our method), resulting in the OS to be 32.
We then upsample the last feature map by 4 times with bilinear interpolation before feeding it into the Encoding Head, noted as Encoding-32-Bilinear.
As shown in Table {{table:eae837dc-ddfd-4800-a13f-30af82c4ce37}} , Encoding-32-Bilinear performs significantly worse than Encoding-8-None, which shows that it's not trivial to replace the dilated convolutions in the backbone of DilatedFCNs.

To show the effectiveness of the proposed JPU, we compare it with other classic upsampling methods, bilinear upsampling and feature pyramid network (FPN) {{cite:101b846}}.
As shown in Table {{table:eae837dc-ddfd-4800-a13f-30af82c4ce37}} , FPN outperforms bilinear interpolation by a large margin.
Even compared with EncNet, FPN achieves comparable performance in both pixAcc and mIoU.
By replacing FPN with our JPU, our method outperforms both FPN and EncNet by more than 1% in mIoU, which achieves the state-of-the-art performance.

To show the generalization ability of the proposed JPU, we replace EncNet with two popular methods in DilatedFCN, namely DeepLabV3 (ASPP Head) {{cite:5378429}} and PSPNet {{cite:fa55743}}.
As shown in Table {{table:eae837dc-ddfd-4800-a13f-30af82c4ce37}} , our methods transformed from DeepLabV3 and PSP outperforms the corresponding original methods consistently.

In Table {{table:eae837dc-ddfd-4800-a13f-30af82c4ce37}} , our method employs ResNet-50 as the backbone without multi-scale evaluation, and the metrics are calculated on 59 classes excluding background by following {{cite:2862f5b}}.
To compare fairly with the state-of-the-art methods, we average the prediction in multiple scales and calculate the metrics among 60 classes including background, which are then reported in Table {{table:21a3f27e-1833-47dd-aa9d-d710ca5b0a5a}} .
With ResNet-50 as the backbone, our method outperforms DeepLabV2 (with COCO pretraining) and RefineNet by a large margin, which employ ResNet-101 and ResNet-152 as the backbone, respectively.
Moreover, our method (ResNet-50) achieves competitive performance compared to EncNet with ResNet-101 as the backbone.
By replacing ResNet-50 with a deeper network ResNet-101, our method gets an additional 1.9% improvement in mIoU, which outperforms EncNet (ResNet-101) and DUpsampling (Xception-71) significantly and achieves the state-of-the-art performance.
Notably, Xception-71 is a much stronger backbone than ResNet-101.
For completeness, we also report the mIoU on 59 classes (w/o background), which is 52.10%(ResNet-50) and 54.03% (ResNet-101).


Task completion ratio and model accuracy/Offloading redundancy, task assignment, beam selection

Model accuracy and energy consumption/Number of local iterations

Sustainable AIGC service provisioning with mobility management

Provision AIGC services via a network of UAVs

MEET: Mobility-Enhanced Edge inTelligence for Smart and Green 6G Networks

Abstract: Edge intelligence is an emerging paradigm for real-time training and inference at the wireless edge, thus enabling mission-critical applications. Accordingly, base stations (BSs) and edge servers (ESs) need to be densely deployed, leading to huge deployment and operation costs, in particular the energy costs. In this article, we propose a new framework called Mobility-Enhanced Edge inTelligence (MEET), which exploits the sensing, communication, computing, and self-powering capabilities of intelligent connected vehicles for the smart and green 6G networks. Specifically, the operators can incorporate infrastructural vehicles as movable BSs or ESs, and schedule them in a more flexible way to align with the communication and computation traffic fluctuations. Meanwhile, the remaining compute resources of opportunistic vehicles are exploited for edge training and inference, where mobility can further enhance edge intelligence by bringing more compute resources, communication opportunities, and diverse data. In this way, the deployment and operation costs are spread over the vastly available vehicles, so that the edge intelligence is realized cost-effectively and sustainably. Furthermore, these vehicles can be either powered by renewable energy to reduce carbon emissions, or charged more flexibly during off-peak hours to cut electricity bills.

UAVs as a Service: Boosting Edge Intelligence for Air-Ground Integrated Networks

Abstract: The air-ground integrated network is a key component of future sixth generation (6G) networks to support seamless and near-instant super-connectivity. There is a pressing need to intelligently provision various services in 6G networks, which however is challenging. To meet this need, in this article, we propose a novel architecture called UaaS (UAVs as a Service) for the air-ground integrated network, featuring UAV as a key enabler to boost edge intelligence with the help of machine learning (ML) techniques. We envision that the proposed UaaS architecture could intelligently provision wireless communication service, edge computing service, and edge caching service by a network of UAVs, making full use of UAVs' flexible deployment and diverse ML techniques. We also conduct a case study where UAVs participate in the model training of distributed ML among multiple terrestrial users, whose result shows that the model training is efficient with a negligible energy consumption of UAVs, compared to the flight energy consumption. Finally, we discuss the challenges and open research issues in the UaaS.

Summary of scenarios, problems, benefits/challenges, and mathematical tools for mobility management.

As summarized in Table {{table:4ef13fff-0e3c-4b1f-8123-02c43bc7e29f}} , here are several KPIs for mobility management in AIGC networks:


Structural Vulnerability of the North American Power Grid

Abstract: The magnitude of the August 2003 blackout affecting the United States has put the challenges of energy transmission and distribution into limelight. Despite all the interest and concerted effort, the complexity and interconnectivity of the electric infrastructure have so far precluded us from understanding why certain events happened. In this paper we study the power grid from a network perspective and determine its ability to transfer power between generators and consumers when certain nodes are disrupted. We find that the power grid is robust to most perturbations, yet disturbances affecting key transmision substations greatly reduce its ability to function. We emphasize that the global properties of the underlying network must be understood as they greatly affect local behavior.

Evaluating North American Electric Grid Reliability Using the Barabasi-Albert Network Model

Abstract: The reliability of electric transmission systems is examined using a scale-free model of network structure and failure propagation. The topologies of the North American eastern and western electric networks are analyzed to estimate their reliability based on the Barabasi-Albert network model. A commonly used power system reliability index is computed using a simple failure propagation model. The results are compared to the values of power system reliability indices previously obtained using standard power system reliability analysis methods, and they suggest that scale-free network models are useful for estimating aggregate electric network reliability.

Electric power grid network databases. Complete networkscontain all substations regardless of voltage.

Although useful for predicting blackouts and finding the critical
components of the power transmission network, the approach above was
limited to simple network topologies which do not correspond to
those of the real power transmission networks. In fact, these
networks have the small-world property, high clustering coefficient
(e.g. {{cite:6da0dd0}}), degree distribution in an exponential
form {{cite:afb3f41}}, {{cite:a7bd515}}, and the “bow-tie”
configuration {{cite:48bd548}}. For a power network of 314,123
nodes, Chassin et al. {{cite:48bd548}} considered any
vertex regardless of voltage (note that the size of the power
networks used in the other studies did not take into account nodes
of small voltages, see Table {{table:36d4c4de-7be9-437c-92da-27d184950161}}  for more power
network data), and showed that the network has a radial form. The
generators are in the center, with transmission substations in the
middle and the load centers at the border, in the so-called
“bow-tie” configuration {{cite:513b91c}}. The degree distribution
was power law with a cutoff term.


Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detection

Abstract: The Gleason scoring system is the primary diagnostic and prognostic tool for prostate cancer. In recent years, with the development of digitisation devices, the use of computer vision techniques for the analysis of biopsies has increased. However, to the best of the authors' knowledge, the development of algorithms to automatically detect individual cribriform patterns belonging to Gleason grade 4 has not yet been studied in the literature. The objective of the work presented in this paper is to develop a deep-learning-based system able to support pathologists in the daily analysis of prostate biopsies. The methodological core of this work is a patch-wise predictive model based on convolutional neural networks able to determine the presence of cancerous patterns. In particular, we train from scratch a simple self-design architecture. The cribriform pattern is detected by retraining the set of filters of the last convolutional layer in the network. From the reconstructed prediction map, we compute the percentage of each Gleason grade in the tissue to feed a multi-layer perceptron which provides a biopsy-level score.mIn our SICAPv2 database, composed of 182 annotated whole slide images, we obtained a Cohen's quadratic kappa of 0.77 in the test set for the patch-level Gleason grading with the proposed architecture trained from scratch. Our results outperform previous ones reported in the literature. Furthermore, this model reaches the level of fine-tuned state-of-the-art architectures in a patient-based four groups cross validation. In the cribriform pattern detection task, we obtained an area under ROC curve of 0.82. Regarding the biopsy Gleason scoring, we achieved a quadratic Cohen's Kappa of 0.81 in the test subset. Shallow CNN architectures trained from scratch outperform current state-of-the-art methods for Gleason grades classification.

UniToPatho, a labeled histopathological dataset for colorectal polyps classification and adenoma dysplasia grading

Abstract: Histopathological characterization of colorectal polyps allows to tailor patients' management and follow up with the ultimate aim of avoiding or promptly detecting an invasive carcinoma. Colorectal polyps characterization relies on the histological analysis of tissue samples to determine the polyps malignancy and dysplasia grade. Deep neural networks achieve outstanding accuracy in medical patterns recognition, however they require large sets of annotated training images. We introduce UniToPatho, an annotated dataset of 9536 hematoxylin and eosin (H&E) stained patches extracted from 292 whole-slide images, meant for training deep neural networks for colorectal polyps classification and adenomas grading. We present our dataset and provide insights on how to tackle the problem of automatic colorectal polyps characterization.

Dataset Description.We report data partitioning as well as statistics for the three used datasets.Size. The size of the patch utilized in the dataset, with each patch performing a series of data augmentations before feature extraction.Num.The average number of patches contained in each WSI in the dataset.SICAPv2. SICAPv2 is a dataset containing prostate histological WSIs with Gleason-grade annotations.We assign the Gleason grade labels to the following five categories: `0+0': 0, `3+3': 1, `3+4': 2, `4+3': 2, `4+4': 3, `3+5': 3, `5+3': 3, `4+5': 4, `5+4': 4, `5+5': 4.UnitoPatho. UnitoPatho is a dataset for the classification of colorectal cancer polyps that contains four categories: Normal tissue, Hyperplastic Polyp, Tubular Adenoma, and Tubulo-Villous Adenoma.TMAs. TMAs is a dataset for classifying pancreatic tissue with three categories: pancreatic ductal adenocarcinoma, chronic pancreatitis, and normal pancreatic tissue.

(1) Dataset Description. We conducted experiments using three different cancer datasets, as shown in Table {{table:4474b4f1-0c67-4235-9b9e-20f720d06dd1}} , including SICAPv2 {{cite:8d6b558}} for prostate cancer, UnitoPatho {{cite:faf1917}} for colorectal cancer, and TMAs {{cite:66c9b1f}} for pancreatic cancer. For UnitoPatho we used the official splitting, while for SICAPv2 and TMAs we used 4-fold cross-validation.
(2) MIL Methods. We selected three classic MIL methods, including AMIL {{cite:97acadb}}, LossAttn {{cite:16f2f98}}, and DSMIL {{cite:a474efa}}.
The training is implemented using the open source code {{cite:ad0d7c9}}.
(3) Feature Extraction. We chose two different encoders for feature extraction, including ResNet18 {{cite:131d006}} and RegNetX_004 {{cite:1377bb0}}. The pre-trained weights for the feature extractor are derived from the timm library {{cite:675d60d}}, and the features are 512- and 384-dimensional, respectively.
(4) Comparison Experiments. We compared with the no augmentation method, pseudo bag construction method used in {{cite:6652967}}, {{cite:5a9d911}}, {{cite:5f04bc7}}, feature augmentation framework with Mixup {{cite:e8e12e5}} which is used in {{cite:476afb8}}, {{cite:bca69f5}}, feature augmentation framework with GAN which is used in {{cite:e3debc1}}, and image-level augmentation methods.
(5) Details of AugDiff.
Our implementation is based on the {{cite:a600514}}. The batch size of the training is 1200, the base learning rate is 5.0e-08, the sampling strategy adopts the DDIM {{cite:73d6442}}, the total steps {{formula:85cb36db-cf39-4eb9-8425-23e61a97fb32}}  is set to 20 or 30, and the number of sampling steps {{formula:368560aa-747b-4b15-8532-6e8781ae5f3e}}  is set to 0.2{{formula:fe070394-9edc-4639-a724-a058c8c22177}}  or 0.4 {{formula:b4749e3d-1918-4fc6-aaf6-9ada8450bd82}} .