nuScenes: A Multimodal Dataset for Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.

The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems

Abstract: Scenario-based testing for the safety validation of highly automated vehicles is a promising approach that is being examined in research and industry. This approach heavily relies on data from real-world scenarios to derive the necessary scenario information for testing. Measurement data should be collected at a reasonable effort, contain naturalistic behavior of road users and include all data relevant for a description of the identified scenarios in sufficient quality. However, the current measurement methods fail to meet at least one of the requirements. Thus, we propose a novel method to measure data from an aerial perspective for scenario-based validation fulfilling the mentioned requirements. Furthermore, we provide a large-scale naturalistic vehicle trajectory dataset from German highways called highD. We evaluate the data in terms of quantity, variety and contained scenarios. Our dataset consists of 16.5 hours of measurements from six locations with 110 000 vehicles, a total driven distance of 45 000 km and 5600 recorded complete lane changes. The highD dataset is available online at: http://www.highD-dataset.com

Argoverse: 3D Tracking and Forecasting With Rich Maps

Abstract: We present Argoverse, a dataset designed to support autonomous vehicle perception tasks including 3D tracking and motion forecasting. Argoverse includes sensor data collected by a fleet of autonomous vehicles in Pittsburgh and Miami as well as 3D tracking annotations, 300k extracted interesting vehicle trajectories, and rich semantic maps. The sensor data consists of 360 degree images from 7 cameras with overlapping fields of view, forward-facing stereo imagery, 3D point clouds from long range LiDAR, and 6-DOF pose. Our 290km of mapped lanes contain rich geometric and semantic metadata which are not currently available in any public dataset. All data is released under a Creative Commons license at Argoverse.org. In baseline experiments, we use map information such as lane direction, driveable area, and ground height to improve the accuracy of 3D object tracking. We use 3D object tracking to mine for more than 300k interesting vehicle trajectories to create a trajectory forecasting benchmark. Motion forecasting experiments ranging in complexity from classical methods (k-NN) to LSTMs demonstrate that using detailed vector maps with lane-level information substantially reduces prediction error. Our tracking and forecasting experiments represent only a superficial exploration of the potential of rich maps in robotic perception. We hope that Argoverse will enable the research community to explore these problems in greater depth.

GridSim: A Vehicle Kinematics Engine for Deep Neuroevolutionary Control in Autonomous Driving

Abstract: Current state of the art solutions in the control of an autonomous vehicle mainly use supervised end-to-end learning, or decoupled perception, planning and action pipelines. Another possible solution is deep reinforcement learning, but such a method requires that the agent interacts with its surroundings in a simulated environment. In this paper we introduce GridSim, which is an autonomous driving simulator engine running a car-like robot architecture to generate occupancy grids from simulated sensors. We use GridSim to study the performance of two deep learning approaches, deep reinforcement learning and driving behavioral learning through genetic algorithms. The deep network encodes the desired behavior in a two elements fitness function describing a maximum travel distance and a maximum forward speed, bounded to a specific interval. The algorithms are evaluated on simulated highways, curved roads and inner-city scenarios, all including different driving limitations.

Performance of state-of-the-art approaches using different datasets and different scenarios. Metric results from other authors represent a two second time frame for future intervals for RMSE. Results using different metric or time frames are not reported.

Trajectory Prediction and Objective Optimisation

A table demonstrating the prediction time window for future trajectories and performance is given in Table {{table:e5bdacd6-8965-4a63-9eed-d4d34d8f84a5}} . It is important to note that a true comparison cannot be made due to the different datasets, as well as different criteria for each work, however this shows that our results are within the correct range for state-of-the-art results.


Weakly Supervised Video Moment Retrieval From Text Queries

Abstract: There have been a few recent methods proposed in text to video moment retrieval using natural language queries, but requiring full supervision during training. However, acquiring a large number of training videos with temporal boundary annotations for each text description is extremely time-consuming and often not scalable. In order to cope with this issue, in this work, we introduce the problem of learning from weak labels for the task of text to video moment retrieval. The weak nature of the supervision is because, during training, we only have access to the video-text pairs rather than the temporal extent of the video to which different text descriptions relate. We propose a joint visual-semantic embedding based framework that learns the notion of relevant segments from video using only video-level sentence descriptions. Specifically, our main idea is to utilize latent alignment between video frames and sentence descriptions using Text-Guided Attention (TGA). TGA is then used during the test phase to retrieve relevant moments. Experiments on two benchmark datasets demonstrate that our method achieves comparable performance to state-of-the-art fully supervised approaches.

WSLLN:Weakly Supervised Natural Language Localization Networks

Abstract: We propose weakly supervised language localization networks (WSLLN) to detect events in long, untrimmed videos given language queries. To learn the correspondence between visual segments and texts, most previous methods require temporal coordinates (start and end times) of events for training, which leads to high costs of annotation. WSLLN relieves the annotation burden by training with only video-sentence pairs without accessing to temporal locations of events. With a simple end-to-end structure, WSLLN measures segment-text consistency and conducts segment selection (conditioned on the text) simultaneously. Results from both are merged and optimized as a video-sentence matching problem. Experiments on ActivityNet Captions and DiDeMo demonstrate that WSLLN achieves state-of-the-art performance.

Regularized Two-Branch Proposal Networks for Weakly-Supervised Moment Retrieval in Videos

Abstract: Video moment retrieval aims to localize the target moment in an video according to the given sentence. The weak-supervised setting only provides the video-level sentence annotations during training. Most existing weak-supervised methods apply a MIL-based framework to develop inter-sample confrontment, but ignore the intra-sample confrontment between moments with semantically similar contents. Thus, these methods fail to distinguish the target moment from plausible negative moments. In this paper, we propose a novel Regularized Two-Branch Proposal Network to simultaneously consider the inter-sample and intra-sample confrontments. Concretely, we first devise a language-aware filter to generate an enhanced video stream and a suppressed video stream. We then design the sharable two-branch proposal module to generate positive proposals from the enhanced stream and plausible negative proposals from the suppressed one for sufficient confrontment. Further, we apply the proposal regularization to stabilize the training process and improve model performance. The extensive experiments show the effectiveness of our method. Our code is released at here.

Weakly Supervised Temporal Adjacent Network for Language Grounding

Abstract: Temporal language grounding (TLG) is a fundamental and challenging problem for vision and language understanding. Existing methods mainly focus on fully supervised setting with temporal boundary labels for training, which, however, suffers expensive cost of annotation. In this work, we are dedicated to weakly supervised TLG, where multiple description sentences are given to an untrimmed video without temporal boundary labels. In this task, it is critical to learn a strong cross-modal semantic alignment between sentence semantics and visual content. To this end, we introduce a novel weakly supervised temporal adjacent network (WSTAN) for temporal language grounding. Specifically, WSTAN learns cross-modal semantic alignment by exploiting temporal adjacent network in a multiple instance learning (MIL) paradigm, with a whole description paragraph as input. Moreover, we integrate a complementary branch into the framework, which explicitly refines the predictions with pseudo supervision from the MIL stage. An additional self-discriminating loss is devised on both the MIL branch and the complementary branch, aiming to enhance semantic discrimination by self-supervising. Extensive experiments are conducted on three widely used benchmark datasets, i.e., ActivityNet-Captions, Charades-STA, and DiDeMo, and the results demonstrate the effectiveness of our approach.

Performance Comparison among weakly supervised localization models on DiDeMo.

Weakly Supervised and Unsupervised Localization Models

Weakly supervised video moment localization approaches are mainly evaluated on the DiDeMo, Charadea-STA, and ActivityNet-Captions datasets. We directly summarized their experimental results from the corresponding papers in Table {{table:d6f012ad-c34f-49fa-93d7-3fd01edb7bef}} , {{table:011b83e9-c18d-4bfb-9f1f-7ed4bccaabf3}} , and {{table:0efa1b75-e3e8-4120-a592-a85f325bbaa7}} .
TGA is the first weakly supervised video moment localization model, considered as the de facto baseline result. Among the two-stage approaches (i.e., WSLLN, LoGAN, and VLANet), LoGAN achieves the best performance on DiDeMo and Charadea-STA, as compared to WSLLN and VLANet. This reflects the importance of learning contextualized visual semantic representations for weakly-supervised moment localization. As to the one-stage models, VCA achieves the best performance in terms of “R(1, 0.7)” on both Charadea-STA and ActivityNet-Captions. This is mainly because enumeration based methods would generate overmuch moment candidates, while they cannot well learn discriminative and robust visual features under video-level supervision. Differently, VCA utilizes the similarity among sentences to mine positive pairs as well as negatives pairs for contrastive learning, therefore learning more discriminative visual features.
Besides, BAR as the first work that extends reinforcement learning to weakly supervised video moment localization also achieves promising performance. Particular, it outperforms TGA and SCAN on Charadea-STA and ActivityNet-Captions.


A Principled Approach to Learning Stochastic Representations for Privacy in Deep Neural Inference

Abstract: INFerence-as-a-Service (INFaaS) in the cloud has enabled the prevalent use of Deep Neural Networks (DNNs) in home automation, targeted advertising, machine vision, etc. The cloud receives the inference request as a raw input, containing a rich set of private information, that can be misused or leaked, possibly inadvertently. This prevalent setting can compromise the privacy of users during the inference phase. This paper sets out to provide a principled approach, dubbed Cloak, that finds optimal stochastic perturbations to obfuscate the private data before it is sent to the cloud. To this end, Cloak reduces the information content of the transmitted data while conserving the essential pieces that enable the request to be serviced accurately. The key idea is formulating the discovery of this stochasticity as an offline gradient-based optimization problem that reformulates a pre-trained DNN (with optimized known weights) as an analytical function of the stochastic perturbations. Using Laplace distribution as a parametric model for the stochastic perturbations, Cloak learns the optimal parameters using gradient descent and Monte Carlo sampling. This set of optimized Laplace distributions further guarantee that the injected stochasticity satisfies the -differential privacy criterion. Experimental evaluations with real-world datasets show that, on average, the injected stochasticity can reduce the information content in the input data by 80.07%, while incurring 7.12% accuracy loss.

TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service

Abstract: Machine learning methods are widely used for a variety of prediction problems. \emph{Prediction as a service} is a paradigm in which service providers with technological expertise and computational resources may perform predictions for clients. However, data privacy severely restricts the applicability of such services, unless measures to keep client data private (even from the service provider) are designed. Equally important is to minimize the amount of computation and communication required between client and server. Fully homomorphic encryption offers a possible way out, whereby clients may encrypt their data, and on which the server may perform arithmetic computations. The main drawback of using fully homomorphic encryption is the amount of time required to evaluate large machine learning models on encrypted data. We combine ideas from the machine learning literature, particularly work on binarization and sparsification of neural networks, together with algorithmic tools to speed-up and parallelize computation using encrypted data.

Secure Face Matching Using Fully Homomorphic Encryption

Abstract: Face recognition technology has demonstrated tremendous progress over the past few years, primarily due to advances in representation learning. As we witness the widespread adoption of these systems, it is imperative to consider the security of face representations. In this paper, we explore the practicality of using a fully homomorphic encryption based framework to secure a database of face templates. This framework is designed to preserve the privacy of users and prevent information leakage from the templates, while maintaining their utility through template matching directly in the encrypted domain. Additionally, we also explore a batching and dimensionality reduction scheme to trade-off face matching accuracy and computational complexity. Experiments on benchmark face datasets (LFW, IJB-A, IJB-B, CASIA) indicate that secure face matching can be practically feasible (16 KB template size and 0.01 sec per match pair for 512-dimensional features from SphereFace) while exhibiting minimal loss in matching performance.

DeepSecure: Scalable Provably-Secure Deep Learning

Abstract: This paper proposes DeepSecure, a novel framework that enables scalable execution of the state-of-the-art Deep Learning (DL) models in a privacy-preserving setting. DeepSecure targets scenarios in which neither of the involved parties including the cloud servers that hold the DL model parameters or the delegating clients who own the data is willing to reveal their information. Our framework is the first to empower accurate and scalable DL analysis of data generated by distributed clients without sacrificing the security to maintain efficiency. The secure DL computation in DeepSecure is performed using Yao's Garbled Circuit (GC) protocol. We devise GC-optimized realization of various components used in DL. Our optimized implementation achieves more than 58-fold higher throughput per sample compared with the best-known prior solution. In addition to our optimized GC realization, we introduce a set of novel low-overhead pre-processing techniques which further reduce the GC overall runtime in the context of deep learning. Extensive evaluations of various DL applications demonstrate up to two orders-of-magnitude additional runtime improvement achieved as a result of our pre-processing methodology. This paper also provides mechanisms to securely delegate GC computations to a third party in constrained embedded settings.

XONN: XNOR-based Oblivious Deep Neural Network Inference

Abstract: Advancements in deep learning enable cloud servers to provide inference-as-a-service for clients. In this scenario, clients send their raw data to the server to run the deep learning model and send back the results. One standing challenge in this setting is to ensure the privacy of the clients' sensitive data. Oblivious inference is the task of running the neural network on the client's input without disclosing the input or the result to the server. This paper introduces XONN, a novel end-to-end framework based on Yao's Garbled Circuits (GC) protocol, that provides a paradigm shift in the conceptual and practical realization of oblivious inference. In XONN, the costly matrix-multiplication operations of the deep learning model are replaced with XNOR operations that are essentially free in GC. We further provide a novel algorithm that customizes the neural network such that the runtime of the GC protocol is minimized without sacrificing the inference accuracy. We design a user-friendly high-level API for XONN, allowing expression of the deep learning model architecture in an unprecedented level of abstraction. Extensive proof-of-concept evaluation on various neural network architectures demonstrates that XONN outperforms prior art such as Gazelle (USENIX Security'18) by up to 7x, MiniONN (ACM CCS'17) by 93x, and SecureML (IEEE S&P'17) by 37x. State-of-the-art frameworks require one round of interaction between the client and the server for each layer of the neural network, whereas, XONN requires a constant round of interactions for any number of layers in the model. XONN is first to perform oblivious inference on Fitnet architectures with up to 21 layers, suggesting a new level of scalability compared with state-of-the-art. Moreover, we evaluate XONN on four datasets to perform privacy-preserving medical diagnosis.

Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applications

Abstract: We present Chameleon, a novel hybrid (mixed-protocol) framework for secure function evaluation (SFE) which enables two parties to jointly compute a function without disclosing their private inputs. Chameleon combines the best aspects of generic SFE protocols with the ones that are based upon additive secret sharing. In particular, the framework performs linear operations in the ring $\mathbb{Z}_{2^l}$ using additively secret shared values and nonlinear operations using Yao's Garbled Circuits or the Goldreich-Micali-Wigderson protocol. Chameleon departs from the common assumption of additive or linear secret sharing models where three or more parties need to communicate in the online phase: the framework allows two parties with private inputs to communicate in the online phase under the assumption of a third node generating correlated randomness in an offline phase. Almost all of the heavy cryptographic operations are precomputed in an offline phase which substantially reduces the communication overhead. Chameleon is both scalable and significantly more efficient than the ABY framework (NDSS'15) it is based on. Our framework supports signed fixed-point numbers. In particular, Chameleon's vector dot product of signed fixed-point numbers improves the efficiency of mining and classification of encrypted data for algorithms based upon heavy matrix multiplications. Our evaluation of Chameleon on a 5 layer convolutional deep neural network shows 133x and 4.2x faster executions than Microsoft CryptoNets (ICML'16) and MiniONN (CCS'17), respectively.

Gazelle: A Low Latency Framework for Secure Neural Network Inference

Abstract:
The growing popularity of cloud-based machine learning raises a natural question about the privacy guarantees that can be provided in such a setting. Our work tackles this problem in the context where a client wishes to classify private images using a convolutional neural network (CNN) trained by a server. Our goal is to build efficient protocols whereby the client can acquire the classification result without revealing their input to the server, while guaranteeing the privacy of the server's neural network.
To this end, we design Gazelle, a scalable and low-latency system for secure neural network inference, using an intricate combination of homomorphic encryption and traditional two-party computation techniques (such as garbled circuits). Gazelle makes three contributions. First, we design the Gazelle homomorphic encryption library which provides fast algorithms for basic homomorphic operations such as SIMD (single instruction multiple data) addition, SIMD multiplication and ciphertext permutation. Second, we implement the Gazelle homomorphic linear algebra kernels which map neural network layers to optimized homomorphic matrix-vector multiplication and convolution routines. Third, we design optimized encryption switching protocols which seamlessly convert between homomorphic and garbled circuit encodings to enable implementation of complete neural network inference.
We evaluate our protocols on benchmark neural networks trained on the MNIST and CIFAR-10 datasets and show that Gazelle outperforms the best existing systems such as MiniONN (ACM CCS 2017) by 20 times and Chameleon (Crypto Eprint 2017/1164) by 30 times in online runtime. Similarly when compared with fully homomorphic approaches like CryptoNets (ICML 2016) we demonstrate three orders of magnitude faster online run-time.

Deep Private-Feature Extraction

Abstract: We present and evaluate Deep Private-Feature Extractor (DPFE), a deep model which is trained and evaluated based on information theoretic constraints. Using the selective exchange of information between a user's device and a service provider, DPFE enables the user to prevent certain sensitive information from being shared with a service provider, while allowing them to extract approved information using their model. We introduce and utilize the log-rank privacy, a novel measure to assess the effectiveness of DPFE in removing sensitive information and compare different models based on their accuracy-privacy tradeoff. We then implement and evaluate the performance of DPFE on smartphones to understand its complexity, resource demands, and efficiency tradeoffs. Our results on benchmark image datasets demonstrate that under moderate resource utilization, DPFE can achieve high accuracy for primary tasks while preserving the privacy of sensitive features.

Categorization of some notable privacy-preserving mechanisms for inference. In this table, NB is short for Naive Bayes, and DT is short for Decision Tree.

As shown in Table {{table:317d16db-1a10-4484-b3a8-98a66252cb1f}}  there are fewer works in the field of inference privacy, compared to training. Inference privacy targets systems that are deployed to offer Inference-as-a-Service. In these cases, the deployed system is assumed to be trained and is not to learn anything new from the data provided by the user. It is only supposed to carry out its designated inference task.
The categorization of literature for inference privacy is similar to training, except that there is one extra group here, named Information-Theoretic (IT) privacy. The works in this group usually offer information-theoretic mathematical or empirical evidence of how their methods operate and help privacy. These works are based on the context-aware privacy definition of Section REF , and they aim at decreasing the information content in the data sent to the service provider for inference so that there is only as much information in the input as needed for the service and not more.


A Survey on the Security of Blockchain Systems

Abstract: Since its inception, the blockchain technology has shown promising application prospects. From the initial cryptocurrency to the current smart contract, blockchain has been applied to many fields. Although there are some studies on the security and privacy issues of blockchain, there lacks a systematic examination on the security of blockchain systems. In this paper, we conduct a systematic study on the security threats to blockchain and survey the corresponding real attacks by examining popular blockchain systems. We also review the security enhancement solutions for blockchain, which could be used in the development of various blockchain systems, and suggest some future directions to stir research efforts into this area.

A Review on the Use of Blockchain for the Internet of Things

Abstract: The paradigm of Internet of Things (IoT) is paving the way for a world, where many of our daily objects will be interconnected and will interact with their environment in order to collect information and automate certain tasks. Such a vision requires, among other things, seamless authentication, data privacy, security, robustness against attacks, easy deployment, and self-maintenance. Such features can be brought by blockchain, a technology born with a cryptocurrency called Bitcoin. In this paper, a thorough review on how to adapt blockchain to the specific needs of IoT in order to develop Blockchain-based IoT (BIoT) applications is presented. After describing the basics of blockchain, the most relevant BIoT applications are described with the objective of emphasizing how blockchain can impact traditional cloud-centered IoT applications. Then, the current challenges and possible optimizations are detailed regarding many aspects that affect the design, development, and deployment of a BIoT application. Finally, some recommendations are enumerated with the aim of guiding future BIoT researchers and developers on some of the issues that will have to be tackled before deploying the next generation of BIoT applications.

Blockchain and IoT Integration: A Systematic Survey

Abstract: The Internet of Things (IoT) refers to the interconnection of smart devices to collect data and make intelligent decisions. However, a lack of intrinsic security measures makes IoT vulnerable to privacy and security threats. With its “security by design,” Blockchain (BC) can help in addressing major security requirements in IoT. BC capabilities like immutability, transparency, auditability, data encryption and operational resilience can help solve most architectural shortcomings of IoT. This article presents a comprehensive survey on BC and IoT integration. The objective of this paper is to analyze the current research trends on the usage of BC-related approaches and technologies in an IoT context. This paper presents the following novelties, with respect to related work: (i) it covers different application domains, organizing the available literature according to this categorization, (ii) it introduces two usage patterns, i.e., device manipulation and data management (open marketplace solution), and (iii) it reports on the development level of some of the presented solutions. We also analyze the main challenges faced by the research community in the smooth integration of BC and IoT, and point out the main open issues and future research directions. Last but not least, we also present a survey about novel uses of BC in the machine economy.

Most cited papers and their domains

We have reported the statistics on top 10 most cited papers, their domain, and the number of citations received till December 2020. The results are shown in Table {{table:529fb340-c78d-4021-9c34-3d287b1d296c}} . It can be seen from that highly cited paper is on the “challenges and opportunities” of the blockchain domain.


To enable prosumers to coordinate their energy usage and prepare the buy and sell orders with the purpose of balancing the demand and supply within the community.

To prevent over voltage and reverse power flow issue due to P2P trading.

Peer-to-Peer Energy Trading With Sustainable User Participation: A Game Theoretic Approach

Abstract: This paper explores the feasibility of social cooperation between prosumers within an energy network in establishing their sustainable participation in peer-to-peer (P2P) energy trading. In particular, a canonical coalition game (CCG) is utilized to propose a P2P energy trading scheme, in which a set of participating prosumers form a coalition group to trade their energy, if there is any, with one another. By exploring the concept of the core of the designed CCG framework, the mid-market rate is utilized as a pricing mechanism of the proposed P2P trading to confirm the stability of the coalition as well as to guarantee the benefit to the prosumers for forming the social coalition. This paper further introduces the motivational psychology models that are relevant to the proposed P2P scheme and it is shown that the outcomes of the proposed P2P energy trading scheme satisfy the discussed models. Consequently, it is proven that the proposed scheme is consumer-centric and has the potential to corroborate sustainable prosumer participation in P2P energy trading. Finally, some numerical examples are provided to demonstrate the beneficial properties of the proposed scheme.

Feasibility Study of Financial P2P Energy Trading in a Grid-tied Power Network

Abstract: This paper studies the applicability of peer-to-peer (P2P) energy trading in a grid-tied network. The main objectives are to understand the impact of the financial P2P energy trading on the network operation, and thus demonstrate the importance of taking various issues related to power network into account while designing a practical P2P trading scheme. To do so, a simple mechanism is developed for energy trading among prosumers without considering any network constraints, as done by many existing studies. Once the trading parameters, such as the energy traded by each prosumer in the P2P market and the price per unit of energy are determined, the developed scheme is tested on a low-voltage (LV) network model to check its feasibility of deployment in a real P2P network. It is shown that although the considered trading scheme is economically beneficial to the participating prosumers compared to the existing incentive mechanisms (such as feed-in-tariff), it could be unfit for real deployment due to violating bus voltage limits during multiple P2P trading executed simultaneously. Further, the grid operator may experience financial losses for compensating the losses during P2P transactions.

Summary of different categories of studies that strive to achieve various objectives in the virtual layer and physical layer platforms.

We note that other than the discussed categories, P2P trading is also conducted to achieve peak load shaving {{cite:b60f762}} and setup virtual power plant {{cite:b4f6565}} to provide ancillary services to the grid. A summary of the existing studies that deal with P2P trading in the virtual layer platform is given in Table {{table:b73ce028-2268-425a-a1f2-ca52fa9a8d34}} .


Secure Federated Matrix Factorization

Abstract: To protect user privacy and meet law regulations, federated (machine) learning is obtaining vast interests in recent years. The key principle of federated learning is training a machine learning model without needing to know each user's personal raw private data. In this paper, we propose a secure matrix factorization framework under the federated learning setting, called FedMF. First, we design a user-level distributed matrix factorization framework where the model can be learned when each user only uploads the gradient information (instead of the raw preference data) to the server. While gradient information seems secure, we prove that it could still leak users' raw data. To this end, we enhance the distributed matrix factorization framework with homomorphic encryption. We implement the prototype of FedMF and test it with a real movie rating dataset. Results verify the feasibility of FedMF. We also discuss the challenges for applying FedMF in practice for future research.

FedPOIRec: Privacy Preserving Federated POI Recommendation with Social Influence

Abstract: With the growing number of Location-Based Social Networks, privacy preserving location prediction has become a primary task for helping users discover new points-of-interest (POIs). Traditional systems consider a centralized approach that requires the transmission and collection of users' private data. In this work, we present FedPOIRec, a privacy preserving federated learning approach enhanced with features from users' social circles for top-$N$ POI recommendations. First, the FedPOIRec framework is built on the principle that local data never leave the owner's device, while the local updates are blindly aggregated by a parameter server. Second, the local recommenders get personalized by allowing users to exchange their learned parameters, enabling knowledge transfer among friends. To this end, we propose a privacy preserving protocol for integrating the preferences of a user's friends after the federated computation, by exploiting the properties of the CKKS fully homomorphic encryption scheme. To evaluate FedPOIRec, we apply our approach into five real-world datasets using two recommendation models. Extensive experiments demonstrate that FedPOIRec achieves comparable recommendation quality to centralized approaches, while the social integration protocol incurs low computation and communication overhead on the user side.

Shared MF: A privacy-preserving recommendation system

Abstract: Matrix factorization is one of the most commonly used technologies in recommendation system. With the promotion of recommendation system in e-commerce shopping, online video and other aspects, distributed recommendation system has been widely promoted, and the privacy problem of multi-source data becomes more and more important. Based on Federated learning technology, this paper proposes a shared matrix factorization scheme called SharedMF. Firstly, a distributed recommendation system is built, and then secret sharing technology is used to protect the privacy of local data. Experimental results show that compared with the existing homomorphic encryption methods, our method can have faster execution speed without privacy disclosure, and can better adapt to recommendation scenarios with large amount of data.

Comparison between different privacy mechanism.

To provide a stronger privacy guarantee, many privacy mechanisms (i.e., pseudo items, homomorphic encryption, differential privacy and secret sharing) have been widely used in FedRS, and the comparison between these mechanisms is shown in Table {{table:3a7f0c82-3be0-47f0-a76b-76aef63408f3}} . Firstly, the main protect objects of these mechanisms are different: pseudo items mechanism is to protect user interaction behaviors, and the rest mechanisms are to protect user ratings. Besides, homomorphic encryption can also integrate data from other participants in a privacy-preserving way. Secondly, homomorphic encryption and secret sharing are both encryption-based mechanisms, and they can protect privacy while keeping accuracy. However, the high computation cost of homomorphic encryption limits it's application in large-scale industrial scenarios. Although the secret sharing mechanism reduces the computation costs, the communication costs increase greatly. Pseudo items and differential privacy mechanisms protect privacy by adding random noise, which has low computation costs and don't bring additional communication costs. But the addition of random noise will inevitably affect model performance to a certain extent.


Fairwashing: the risk of rationalization

Abstract: Black-box explanation is the problem of explaining how a machine learning model -- whose internal logic is hidden to the auditor and generally complex -- produces its outcomes. Current approaches for solving this problem include model explanation, outcome explanation as well as model inspection. While these techniques can be beneficial by providing interpretability, they can be used in a negative manner to perform fairwashing, which we define as promoting the false perception that a machine learning model respects some ethical values. In particular, we demonstrate that it is possible to systematically rationalize decisions taken by an unfair black-box model using the model explanation as well as the outcome explanation approaches with a given fairness metric. Our solution, LaundryML, is based on a regularized rule list enumeration algorithm whose objective is to search for fair rule lists approximating an unfair black-box model. We empirically evaluate our rationalization technique on black-box models trained on real-world datasets and show that one can obtain rule lists with high fidelity to the black-box model while being considerably less unfair at the same time.

Exacerbating Algorithmic Bias through Fairness Attacks

Abstract: Algorithmic fairness has attracted significant attention in recent years, with many quantitative measures suggested for characterizing the fairness of different machine learning algorithms. Despite this interest, the robustness of those fairness measures with respect to an intentional adversarial attack has not been properly addressed. Indeed, most adversarial machine learning has focused on the impact of malicious attacks on the accuracy of the system, without any regard to the system's fairness. We propose new types of data poisoning attacks where an adversary intentionally targets the fairness of a system. Specifically, we propose two families of attacks that target fairness measures. In the anchoring attack, we skew the decision boundary by placing poisoned points near specific target points to bias the outcome. In the influence attack on fairness, we aim to maximize the covariance between the sensitive attributes and the decision outcome and affect the fairness of the model. We conduct extensive experiments that indicate the effectiveness of our proposed attacks.

Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning

Abstract: Deep neural networks (DNNs) are increasingly used in real-world applications (e.g. facial recognition). This has resulted in concerns about the fairness of decisions made by these models. Various notions and measures of fairness have been proposed to ensure that a decision-making system does not disproportionately harm (or benefit) particular subgroups of the population. In this paper, we argue that traditional notions of fairness that are only based on models' outputs are not sufficient when the model is vulnerable to adversarial attacks. We argue that in some cases, it may be easier for an attacker to target a particular subgroup, resulting in a form of robustness bias. We show that measuring robustness bias is a challenging task for DNNs and propose two methods to measure this form of bias. We then conduct an empirical study on state-of-the-art neural networks on commonly used real-world datasets such as CIFAR-10, CIFAR-100, Adience, and UTKFace and show that in almost all cases there are subgroups (in some cases based on sensitive attributes like race, gender, etc) which are less robust and are thus at a disadvantage. We argue that this kind of bias arises due to both the data distribution and the highly complex nature of the learned decision boundary in the case of DNNs, thus making mitigation of such biases a non-trivial task. Our results show that robustness bias is an important criterion to consider while auditing real-world systems that rely on DNNs for decision making. Code to reproduce all our results can be found here: https://github.com/nvedant07/Fairness-Through-Robustness

Characterizing the risk of fairwashing

Abstract: Fairwashing refers to the risk that an unfair black-box model can be explained by a fairer model through post-hoc explanation manipulation. In this paper, we investigate the capability of fairwashing attacks by analyzing their fidelity-unfairness trade-offs. In particular, we show that fairwashed explanation models can generalize beyond the suing group (i.e., data points that are being explained), meaning that a fairwashed explainer can be used to rationalize subsequent unfair decisions of a black-box model. We also demonstrate that fairwashing attacks can transfer across black-box models, meaning that other black-box models can perform fairwashing without explicitly using their predictions. This generalization and transferability of fairwashing attacks imply that their detection will be difficult in practice. Finally, we propose an approach to quantify the risk of fairwashing, which is based on the computation of the range of the unfairness of high-fidelity explainers.

Adversarial Inter-Group Link Injection Degrades the Fairness of Graph Neural Networks

Abstract: We present evidence for the existence and effectiveness of adversarial attacks on graph neural networks (GNNs) that aim to degrade fairness. These attacks can disadvantage a particular subgroup of nodes in GNN-based node classification, where nodes of the underlying network have sensitive attributes, such as race or gender. We conduct qualitative and experimental analyses explaining how adversarial link injection impairs the fairness of GNN predictions. For example, an attacker can compromise the fairness of GNN-based node classification by injecting adversarial links between nodes belonging to opposite subgroups and opposite class labels. Our experiments on empirical datasets demonstrate that adversarial fairness attacks can significantly degrade the fairness of GNN predictions (attacks are effective) with a low perturbation rate (attacks are efficient) and without a significant drop in accuracy (attacks are deceptive). This work demonstrates the vulnerability of GNN models to adversarial fairness attacks. We hope our findings raise awareness about this issue in our community and lay a foundation for the future development of GNN models that are more robust to such attacks.

Exploiting Fairness to Enhance Sensitive Attributes Reconstruction

Abstract: In recent years, a growing body of work has emerged on how to learn machine learning models under fairness constraints, often expressed with respect to some sensitive attributes. In this work, we consider the setting in which an adversary has black-box access to a target model and show that information about this model's fairness can be exploited by the adversary to enhance his reconstruction of the sensitive attributes of the training data. More precisely, we propose a generic reconstruction correction method, which takes as input an initial guess made by the adversary and corrects it to comply with some user-defined constraints (such as the fairness information) while minimizing the changes in the adversary's guess. The proposed method is agnostic to the type of target model, the fairness-aware learning method as well as the auxiliary knowledge of the adversary. To assess the applicability of our approach, we have conducted a thorough experimental evaluation on two state-of-the-art fair learning methods, using four different fairness metrics with a wide range of tolerances and with three datasets of diverse sizes and sensitive attributes. The experimental results demonstrate the effectiveness of the proposed approach to improve the reconstruction of the sensitive attributes of the training set.

Related work concerning adversarial attacks on fairness metrics of machine learning models. We abbreviate the following: data (D), model (M), image (I), tabular (T), graph (Gr), neural network (N), black-box (B), group (G). Appendix  lists other abbreviations.

Closely related to adversarial attacks on explanations are attacks on machine learning fairness metrics, e.g. predictive equality {{cite:029d5d2}} and (statistical) demographic parity {{cite:5d07523}}. Intuitively, algorithms targeting model predictions and accuracy can be applied to manipulate other functions of the model output as well. Table {{table:3d8c9d59-c3b7-4180-ba0c-6c677b6d5061}}  lists a representative set of adversarial attacks on group fairness metrics, with the corresponding strategy of changing data {{cite:94071a9}}, the model {{cite:912a49a}}, {{cite:5952c82}}, or jointly changing data and the model {{cite:f581029}}, {{cite:e0c8b9d}}, {{cite:2e607ab}}.


Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

Remote Sensing Image Scene Classification: Benchmark and State of the Art

Abstract: Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various data sets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning data sets and methods for scene classification is still lacking. In addition, almost all existing data sets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale data set, termed “NWPU-RESISC45,” which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This data set contains 31 500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 1) is large-scale on the scene classes and the total image number; 2) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion; and 3) has high within-class diversity and between-class similarity. The creation of this data set will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed data set, and the results are reported as a useful baseline for future research.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Abstract: When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover short-comings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

Details on the different benchmarks used in the study, including the number of classes, tasks, and target domain.

Where we assume the user supplies the target domain and task description. For examples of different dataset's domain and task, see Tab. {{table:f16e481e-c72d-46fc-a08b-1b4b83d7acfe}} .

The proposed LOVM benchmark comprises of 23 datasets, which were selected to maximize diversity. Specifically, these datasets vary in the number of classes (2 to 1000), their target tasks, and domains.
The benchmark encompasses a comprehensive range of tasks such as classification, scene understanding, geolocalization, object counting, and more, with the goal of rendering it extensively applicable across many applications.
Further, the datasets span diverse domains, including natural, satellite, text, and medical images (See Tab. {{table:f16e481e-c72d-46fc-a08b-1b4b83d7acfe}}  for a comprehensive account of the datasets and their source).
To ensure maximal compatibility, we have opted for tasks that permit the utilization of the same VLM architecture, precluding any requisite alterations or additional training.
This approach necessitated the exclusion of tasks such as segmentation and object detection, which mandate additional training modules, introducing extraneous noise while evaluating VLM performance. However, it is worth noting that previous transferability works have shown that these approaches may generalize to more complex applications such as semantic segmentation {{cite:8d550b2}}, {{cite:c72904d}}.


Surveys the existing works on MEc for Industrial Internet.

Explores dependability and deployment challenges in edge computing. Considers dependability in a wider meaning that includes security.

Discusses the feasibility and potential of providing edge computing services with latency and reliability guarantees.

Discusses the high reliability as typical industrial requirement.

Presents resiliency challenges, which includes new failures modes, network impact, limited fail-over options, multi-tenancy support, and interoperability.

Overviews the challenges and the enablers for realizing hugh reliability in wireless edge computing.

A Survey of Mobile Edge Computing in the Industrial Internet

Abstract: With the advent of a new round of the Industrial Revolution, the Industrial Internet will carry the convergence of heterogeneous network and the dynamic reconfiguration of industrial equipment. In order to further provide higher performance of network capabilities, the Industrial Internet has experienced unprecedented growth while facing enormous challenges from the actual needs of industrial networks. The typical scenarios in industrial applications, combined with the technical advantages of mobile edge computing, are described in view of the low latency, high bandwidth and high reliability demanded by the Industrial Internet in the new era. The key technologies of mobile edge computing for the Industrial Internet have been outlined in this treatise, whose feasibility and importance are demonstrated by typical industrial applications that have been deployed. As combined with the development trend of the Industrial Internet, this paper summarizes the existing work and discusses the future research direction of key technologies of mobile edge computing for the Industrial Internet.

Dependability in Edge Computing

Abstract: Edge computing is the practice of placing computing resources at the edges of the Internet in close proximity to devices and information sources. This, much like a cache on a CPU, increases bandwidth and reduces latency for applications but at a potential cost of dependability and capacity. This is because these edge devices are often not as well maintained, dependable, powerful, or robust as centralized server-class cloud resources. This article explores dependability and deployment challenges in the field of edge computing, what aspects are solvable with today's technology, and what aspects call for new solutions. The first issue addressed is failures, both hard (crash, hang, etc.) and soft (performance-related), and real-time constraint violation. In this domain, edge computing bolsters real-time system capacity through reduced end-to-end latency. However, much like cache misses, overloaded or malfunctioning edge computers can drive latency beyond tolerable limits. Second, decentralized management and device tampering can lead to chain of trust and security or privacy violations. Authentication, access control, and distributed intrusion detection techniques have to be extended from current cloud deployments and need to be customized for the edge ecosystem. The third issue deals with handling multi-tenancy in the typically resource-constrained edge devices and the need for standardization to allow for interoperability across vendor products. We explore the key challenges in each of these three broad issues as they relate to dependability of edge computing and then hypothesize about promising avenues of work in this area.

Wireless Edge Computing With Latency and Reliability Guarantees

Abstract: Edge computing is an emerging concept based on distributed computing, storage, and control services closer to end network nodes. Edge computing lies at the heart of the fifth-generation (5G) wireless systems and beyond. While the current state-of-the-art networks communicate, compute, and process data in a centralized manner (at the cloud), for latency and compute-centric applications, both radio access and computational resources must be brought closer to the edge, harnessing the availability of computing and storage-enabled small cell base stations in proximity to the end devices. Furthermore, the network infrastructure must enable a distributed edge decision-making service that learns to adapt to the network dynamics with minimal latency and optimize network deployment and operation accordingly. This paper will provide a fresh look to the concept of edge computing by first discussing the applications that the network edge must provide, with a special emphasis on the ensuing challenges in enabling ultrareliable and low-latency edge computing services for mission-critical applications such as virtual reality (VR), vehicle-to-everything (V2X), edge artificial intelligence (AI), and so on. Furthermore, several case studies where the edge is key are explored followed by insights and prospect for future work.

Academic Surveys Related to MEC Dependability

Table {{table:a621d9c3-acb5-432a-bfeb-ef9f24db2540}}  lists recent surveys that address dependability aspects in MEC.
As for security, not many papers are fully dedicated to MEC but consider it together with other technologies such as cloud or fog.
A survey on dependability on MEC does not exists. A few papers are mainly focused on dependability aspects {{cite:7219731}}, {{cite:9775329}}, {{cite:1188458}} and others have parts dedicated to dependability aspects {{cite:5454e73}}, {{cite:43f0c42}}, {{cite:0cc80b4}} or shared with other aspects {{cite:d346983}}, {{cite:9c7b155}}, {{cite:72d4ea2}}, {{cite:b8c3a87}}. The other papers just mention it or use it as requirement {{cite:9c7b155}}, {{cite:43f0c42}}, property {{cite:9455e32}}, benefit {{cite:5efcfa0}}, or challenge {{cite:0cc80b4}}, {{cite:e376f0f}}.


Masked Face Recognition Challenge: The InsightFace Track Report

Abstract: During the COVID-19 coronavirus epidemic, almost everyone wears a facial mask, which poses a huge challenge to deep face recognition. In this workshop, we organize Masked Face Recognition (MFR) challenge1 and focus on bench-marking deep face recognition methods under the existence of facial masks. In the MFR challenge, there are two main tracks: the InsightFace track and the WebFace260M track [38]. For the InsightFace track, we manually collect a large-scale masked face test set with 7K identities. In addition, we also collect a children test set including 14K identities and a multi-racial test set containing 242K identities. By using these three test sets, we build up an online model testing system, which can give a comprehensive evaluation of face recognition models. To avoid data privacy problems, no test image is released to the public. As the challenge is still under-going, we will keep on updating the top-ranked solutions as well as this report on the arxiv.

Deep Learning Face Attributes in the Wild

Abstract: Predicting face attributes in the wild is challenging due to complex face variations. We propose a novel deep learning framework for attribute prediction in the wild. It cascades two CNNs, LNet and ANet, which are fine-tuned jointly with attribute tags, but pre-trained differently. LNet is pre-trained by massive general object categories for face localization, while ANet is pre-trained by massive face identities for attribute prediction. This framework not only outperforms the state-of-the-art with a large margin, but also reveals valuable facts on learning face representation. (1) It shows how the performances of face localization (LNet) and attribute prediction (ANet) can be improved by different pre-training strategies. (2) It reveals that although the filters of LNet are fine-tuned only with image-level attribute tags, their response maps over entire images have strong indication of face locations. This fact enables training LNet for face localization with only image-level annotations, but without face bounding boxes or landmarks, which are required by all attribute recognition works. (3) It also demonstrates that the high-level hidden neurons of ANet automatically discover semantic concepts after pre-training with massive face identities, and such concepts are significantly enriched after fine-tuning with attribute tags. Each attribute can be well explained with a sparse linear combination of these concepts.

DigiFace-1M: 1 Million Digital Face Images for Face Recognition

Abstract: State-of-the-art face recognition models show impressive accuracy, achieving over 99.8% on Labeled Faces in the Wild (LFW) dataset. Such models are trained on large-scale datasets that contain millions of real human face images collected from the internet. Web-crawled face images are severely biased (in terms of race, lighting, makeup, etc) and often contain label noise. More importantly, the face images are collected without explicit consent, raising ethical concerns. To avoid such problems, we introduce a large-scale synthetic dataset for face recognition, obtained by rendering digital faces using a computer graphics pipeline1. We first demonstrate that aggressive data augmentation can significantly reduce the synthetic-to-real domain gap. Having full control over the rendering pipeline, we also study how each attribute (e.g., variation in facial pose, accessories and textures) affects the accuracy. Compared to Syn-Face, a recent method trained on GAN-generated synthetic faces, we reduce the error rate on LFW by 52.5% (accuracy from 91.93% to 96.17%). By fine-tuning the network on a smaller number of real face images that could reason-ably be obtained with consent, we achieve accuracy that is comparable to the methods trained on millions of real face images.

Statistics of train and test data.

Testing Dataset. We performed extensive experiments on various benchmarks with different emphases, summarized in Table {{table:61e0b6bc-b702-45f3-984e-f5b58c52083b}} .
We evaluated fair face recognition on 3 datasets with attribute labels:
(1) MFR {{cite:0b15e2a}} is a large-scale non-celebrity dataset with racial attribute annotations. It serves as a challenging benchmark to test model generalization in different racial groups.
(2) RFW {{cite:ad6673f}} is a race-balanced dataset constructed from the original MS-Celeb-1M {{cite:c392dcb}}. We followed  {{cite:7317a2f}} to exclude the overlapping identities from RFW with the training dataset.
(3) CelebA {{cite:bef24d6}} provides ground-truth labels for fine-grained attributes , gender, hair color and style. We combined data from validation and test splits to evaluate our models.
In addition, we used 2 datasets to evaluate conventional face recognition.
(4) IJB-B and IJB-C {{cite:5ad857e}}, {{cite:20a6a60}} are large-scale datasets with images and continuous video frames collected under unconstrained scenarios.
(5) DigiFace-1M {{cite:77a7193}} is the latest synthetic dataset of realistic face images. It renders each identity under large variations, , hair density, makeup, face-wear, and expression. Hence we adopted it to challenge the generalization ability of models. We used the first 1K identities for testing, where each identity has 72 images.


YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors

Abstract: Real-time object detection is one of the most important research topics in computer vision. As new approaches regarding architecture optimization and training optimization are continually being developed, we have found two research topics that have spawned when dealing with these latest state-of-the-art methods. To address the topics, we propose a trainable bag-of-freebies oriented solution. We combine the flexible and efficient training tools with the proposed architecture and the compound scaling method. YOLOv7 surpasses all known object detectors in both speed and accuracy in the range from 5 FPS to 120 FPS and has the highest accuracy 56.8% AP among all known real-time object detectors with 30 FPS or higher on GPU V100. Source code is released in https://github.com/WongKinYiu/yolov7.

CSPNet: A New Backbone that can Enhance Learning Capability of CNN

Abstract: Neural networks have enabled state-of-the-art approaches to achieve incredible results on computer vision tasks such as object detection. However, such success greatly relies on costly computation resources, which hinders people with cheap devices from appreciating the advanced technology. In this paper, we propose Cross Stage Partial Network (CSPNet) to mitigate the problem that previous works require heavy inference computations from the network architecture perspective. We attribute the problem to the duplicate gradient information within network optimization. The proposed networks respect the variability of the gradients by integrating feature maps from the beginning and the end of a network stage, which, in our experiments, reduces computations by 20% with equivalent or even superior accuracy on the ImageNet dataset, and significantly outperforms state-of-the-art approaches in terms of AP50 on the MS COCO object detection dataset. The CSPNet is easy to implement and general enough to cope with architectures based on ResNet, ResNeXt, and DenseNet.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Abstract:
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.
To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.4% top-1 / 97.1% top-5 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at this https URL.

Scaled-YOLOv4: Scaling Cross Stage Partial Network

Abstract: We show that the YOLOv4 object detection neural network based on the CSP approach, scales both up and down and is applicable to small and large networks while maintaining optimal speed and accuracy. We propose a network scaling approach that modifies not only the depth, width, resolution, but also structure of the network. YOLOv4-large model achieves state-of-the-art results: 55.5% AP (73.4% AP50) for the MS COCO dataset at a speed of ~ 16 FPS on Tesla V100, while with the test time augmentation, YOLOv4-large achieves 56.0% AP (73.3 AP50). To the best of our knowledge, this is currently the highest accuracy on the COCO dataset among any published work. The YOLOv4-tiny model achieves 22.0% AP (42.0% AP50) at a speed of ~443 FPS on RTX 2080Ti, while by using TensorRT, batch size = 4 and FP16-precision the YOLOv4-tiny achieves 1774 FPS.

Comparison of state-of-the-art real-time object detectors.

Comparisons with State-of-The-Art real-time Models

As illustrated in Table {{table:84102dd2-aee7-4857-8e1d-f37883bbc58c}} , a comparative analysis was conducted between the PRB-FPN equipped with various backbones {{cite:f3bb9eb}}, {{cite:69e72f7}}, {{cite:28cc38d}}, {{cite:6502caa}} and the existing SoTA methods {{cite:fca3b2d}}, {{cite:5e14dcf}}, {{cite:28cc38d}}, {{cite:26f3196}}. Noticeably, PRB-FPN with a scaled P6 backbone surpasses all contemporary methods, thereby establishing a new SoTA on the real-time object detection COCO benchmark {{cite:bac6080}}. Furthermore, the P5 variant of PRB-FPN, incorporating MSPNet {{cite:6502caa}}, {{cite:69e72f7}}, also exhibits superior performance over the YOLOv7-X {{cite:28cc38d}} equipped with an ELAN {{cite:28cc38d}} backbone, while maintaining a comparable computational cost.


Machine learning based interatomic potential for amorphous carbon

Abstract: We introduce a Gaussian approximation potential (GAP) for atomistic simulations of liquid and amorphous elemental carbon. Based on a machine learning representation of the density-functional theory (DFT) potential-energy surface, such interatomic potentials enable materials simulations with close-to DFT accuracy but at much lower computational cost. We first determine the maximum accuracy that any finite-range potential can achieve in carbon structures; then, using a hierarchical set of two-, three-, and many-body structural descriptors, we construct a GAP model that can indeed reach the target accuracy. The potential yields accurate energetic and structural properties over a wide range of densities; it also correctly captures the structure of the liquid phases, at variance with a state-of-the-art empirical potential. Exemplary applications of the GAP model to surfaces of “diamondlike” tetrahedral amorphous carbon ($\textit{ta}$-C) are presented, including an estimate of the amorphous material’s surface energy and simulations of high-temperature surface reconstructions (“graphitization”). The presented interatomic potential appears to be promising for realistic and accurate simulations of nanoscale amorphous carbon structures.

An Accurate and Transferable Machine Learning Potential for Carbon

Abstract: We present an accurate machine learning (ML) model for atomistic simulations of carbon, constructed using the Gaussian approximation potential (GAP) methodology. The potential, named GAP-20, describes the properties of the bulk crystalline and amorphous phases, crystal surfaces and defect structures with an accuracy approaching that of direct ab initio simulation, but at a significantly reduced cost. We combine structural databases for amorphous carbon and graphene, which we extend substantially by adding suitable configurations, for example, for defects in graphene and other nanostructures. The final potential is fitted to reference data computed using the optB88-vdW density functional theory (DFT) functional. Dispersion interactions, which are crucial to describe multilayer carbonaceous materials, are therefore implicitly included. We additionally account for long-range dispersion interactions using a semianalytical two-body term and show that an improved model can be obtained through an optimisation of the many-body smooth overlap of atomic positions (SOAP) descriptor. We rigorously test the potential on lattice parameters, bond lengths, formation energies and phonon dispersions of numerous carbon allotropes. We compare the formation energies of an extensive set of defect structures, surfaces and surface reconstructions to DFT reference calculations. The present work demonstrates the ability to combine, in the same ML model, the previously attained flexibility required for amorphous carbon [Phys. Rev. B, 95, 094203, (2017)] with the high numerical accuracy necessary for crystalline graphene [Phys. Rev. B, 97, 054303, (2018)], thereby providing an interatomic potential that will be applicable to a wide range of applications concerning diverse forms of bulk and nanostructured carbon.

GPUMD: A package for constructing accurate machine-learned potentials and performing highly efficient atomistic simulations

Abstract: We present our latest advancements of machine-learned potentials (MLPs) based on the neuroevolution potential (NEP) framework introduced in [Fan et al., Phys. Rev. B 104, 104309 (2021)] and their implementation in the open-source package GPUMD. We increase the accuracy of NEP models both by improving the radial functions in the atomic-environment descriptor using a linear combination of Chebyshev basis functions and by extending the angular descriptor with some four-body and five-body contributions as in the atomic cluster expansion approach. We also detail our efficient implementation of the NEP approach in graphics processing units as well as our workflow for the construction of NEP models, and we demonstrate their application in large-scale atomistic simulations. By comparing to state-of-the-art MLPs, we show that the NEP approach not only achieves above-average accuracy but also is far more computationally efficient. These results demonstrate that the GPUMD package is a promising tool for solving challenging problems requiring highly accurate, large-scale atomistic simulations. To enable the construction of MLPs using a minimal training set, we propose an active-learning scheme based on the latent space of a pre-trained NEP model. Finally, we introduce three separate Python packages, GPYUMD, CALORINE, and PYNEP, which enable the integration of GPUMD into Python workflows.

Atomic cluster expansion for quantum-accurate large-scale simulations of carbon

Abstract: We present an atomic cluster expansion (ACE) for carbon that improves over available classical and machine learning potentials. The ACE is parameterized from an exhaustive set of important carbon structures at extended volume and energy range, computed using density functional theory (DFT). Rigorous validation reveals that ACE predicts accurately a broad range of properties of both crystalline and amorphous carbon phases while being several orders of magnitude more computationally efficient than available machine learning models. We demonstrate the predictive power of ACE on three distinct applications, brittle crack propagation in diamond, evolution of amorphous carbon structures at different densities and quench rates and nucleation and growth of fullerene clusters under high pressure and temperature conditions.

Machine Learning a General-Purpose Interatomic Potential for Silicon

Abstract: The success of first principles electronic structure calculation for predictive modeling in chemistry, solid state physics, and materials science is constrained by the limitations on simulated length and time scales due to computational cost and its scaling. Techniques based on machine learning ideas for interpolating the Born-Oppenheimer potential energy surface without explicitly describing electrons have recently shown great promise, but accurately and efficiently fitting the physically relevant space of configurations has remained a challenging goal. Here we present a Gaussian Approximation Potential for silicon that achieves this milestone, accurately reproducing density functional theory reference results for a wide range of observable properties, including crystal, liquid, and amorphous bulk phases, as well as point, line, and plane defects. We demonstrate that this new potential enables calculations that would be extremely expensive with a first principles electronic structure method, such as finite temperature phase boundary lines, self-diffusivity in the liquid, formation of the amorphous by slow quench, and dynamic brittle fracture. We show that the uncertainty quantification inherent to the Gaussian process regression framework gives a qualitative estimate of the potential's accuracy for a given atomic configuration. The success of this model shows that it is indeed possible to create a useful machine-learning-based interatomic potential that comprehensively describes a material, and serves as a template for the development of such models in the future.

Neuroevolution machine learning potentials: Combining high accuracy and low cost in atomistic simulations and application to heat transport

Abstract: We develop a neuroevolution-potential (NEP) framework for generating neural network based machine-learning potentials. They are trained using an evolutionary strategy for performing large-scale molecular dynamics (MD) simulations. A descriptor of the atomic environment is constructed based on Chebyshev and Legendre polynomials. The method is implemented in graphic processing units within the open-source GPUMD package, which can attain a computational speed over $10^7$ atom-step per second using one Nvidia Tesla V100. Furthermore, per-atom heat current is available in NEP, which paves the way for efficient and accurate MD simulations of heat transport in materials with strong phonon anharmonicity or spatial disorder, which usually cannot be accurately treated either with traditional empirical potentials or with perturbative methods.

A Gaussian Approximation Potential for Amorphous Si:H

Abstract: Hydrogenation of amorphous silicon (a-Si:H) is critical for reducing defect densities, passivating mid-gap states and surfaces, and improving photoconductivity in silicon-based electro-optical devices. Modelling the atomic scale structure of this material is critical to understanding these processes, which in turn is needed to describe c-Si/a-Si:H heterjunctions that are at the heart of the modern solar cells with world record efficiency. Density functional theory (DFT) studies achieve the required high accuracy but are limited to moderate system sizes a hundred atoms or so by their high computational cost. Simulations of amorphous materials in particular have been hindered by this high cost because large structural models are required to capture the medium range order that is characteristic of such materials. Empirical potential models are much faster, but their accuracy is not sufficient to correctly describe the frustrated local structure. Data driven, "machine learned" interatomic potentials have broken this impasse, and have been highly successful in describing a variety of amorphous materials in their elemental phase. Here we extend the Gaussian approximation potential (GAP) for silicon by incorporating the interaction with hydrogen, thereby significantly improving the degree of realism with which amorphous silicon can be modelled. We show that our Si:H GAP enables the simulation of hydrogenated silicon with an accuracy very close to DFT, but with computational expense and run times reduced by several orders of magnitude for large structures. We demonstrate the capabilities of the Si:H GAP by creating models of hydrogenated liquid and amorphous silicon, and showing that their energies, forces and stresses are in excellent agreement with DFT results, and their structure as captured by bond and angle distributions, with both DFT and experiments.

List of some of the publicly available MLPs to simulate disordered C and Si, together withpractical information. “Code” refers to the computer software which can be used to run asimulation with the corresponding MLP. GP stands for “general purpose”, i.e., it refers toan MLP which is not tailored exclusively to simulate a disordered phase but can be reliablyused for that purpose. Whenever more than one version of the MLP exists, we give the referenceto the latest one (i.e., on the table v2 refers to version 2 of a given MLP).

Table {{table:ae6e9b16-e1f5-4416-8d9c-25999e95a18c}}  provides a non-comprehensive list of available MLPs able to simulate a-C and a-Si,
together with their ML “flavor” and which code(s) they can be used with. These potentials are
mostly of the GAP flavor, since the GAP community has been the most active at simulating a-C and a-Si
among the different MLP developers and users base.