Highlight of the main attacks against RF EH networks.

Overview of few research areas related to security in SWIPT networks.

Literature review not provided. Weaknesses, countermeasures, and research challenges related to the security domain are not discussed.

Literature review not provided. Focus on Information Secrecy only. Defensive technologies, weaknesses, countermeasures, and research challenges related to the security domain are not discussed.

Safe and Secure Wireless Power Transfer Networks: Challenges and Opportunities in RF-Based Systems

Abstract: RF-based wireless power transfer networks (WPTNs) are deployed to transfer power to embedded devices over the air via RF waves. Up until now, a considerable amount of effort has been devoted by researchers to design WPTNs that maximize several objectives such as harvested power, energy outage and charging delay. However, inherent security and safety issues are generally overlooked and these need to be solved if WPTNs are to be become widespread. This article focuses on safety and security problems related WPTNs and highlight their cruciality in terms of efficient and dependable operation of RF-based WPTNs. We provide a overview of new research opportunities in this emerging domain.

Secrecy wireless information and power transfer: challenges and opportunities

Abstract: Wireless information and power transfer (WIPT) enables more sustainable and resilient communications due to the fact that it avoids frequent battery charging and replacement. However, it also suffers from possible information interception due to the open nature of wireless channels. Compared to traditional secure communications, secrecy wireless information and power transfer (SWIPT) has several distinct characteristic features. Wireless power transfer may increase vulnerability to eavesdropping, since a power receiver as a potential eavesdropper usually has a shorter access distance than an information receiver. On the other hand, wireless power transfer can be exploited to enhance wireless security. This article reviews the security issues in various SWIPT scenarios, with an emphasis on revealing the challenges and opportunities for implementing SWIPT. Furthermore, we provide a survey on a variety of physical layer security techniques to improve secrecy performance. In particular, we propose to use massive MIMO techniques to enhance power transfer efficiency and secure information transmission simultaneously. Finally, we discuss several potential research directions to further enhance the security in SWIPT systems.

Main addressed topics and differences between surveys touching EH Network Security

Security issues in EH networks have been quickly touched by few position papers and surveys available in the literature. An overview of their main features and differences with this contribution is provided in Tab. {{table:790ad8fb-0e89-42f5-86e6-eb8b24bc59dd}} .


3D Hand Pose Tracking and Estimation Using Stereo Matching

Abstract: 3D hand pose tracking/estimation will be very important in the next generation of human-computer interaction. Most of the currently available algorithms rely on low-cost active depth sensors. However, these sensors can be easily interfered by other active sources and require relatively high power consumption. As a result, they are currently not suitable for outdoor environments and mobile devices. This paper aims at tracking/estimating hand poses using passive stereo which avoids these limitations. A benchmark with 18,000 stereo image pairs and 18,000 depth images captured from different scenarios and the ground-truth 3D positions of palm and finger joints (obtained from the manual label) is thus proposed. This paper demonstrates that the performance of the state-of-the art tracking/estimation algorithms can be maintained with most stereo matching algorithms on the proposed benchmark, as long as the hand segmentation is correct. As a result, a novel stereo-based hand segmentation algorithm specially designed for hand tracking/estimation is proposed. The quantitative evaluation demonstrates that the proposed algorithm is suitable for the state-of-the-art hand pose tracking/estimation algorithms and the tracking quality is comparable to the use of active depth sensors under different challenging scenarios.

Learning to Estimate 3D Hand Pose from Single RGB Images

Abstract: Low-cost consumer depth cameras and deep learning have enabled reasonable 3D hand pose estimation from single depth images. In this paper, we present an approach that estimates 3D hand pose from regular RGB images. This task has far more ambiguities due to the missing depth information. To this end, we propose a deep network that learns a network-implicit 3D articulation prior. Together with detected keypoints in the images, this network yields good estimates of the 3D pose. We introduce a large scale 3D hand pose dataset based on synthetic hand models for training the involved networks. Experiments on a variety of test sets, including one on sign language recognition, demonstrate the feasibility of 3D hand pose estimation on single color images.

Public datasets used in our experiments.

We carried out experiments on both 3D and 2D hand pose estimation from RGB images.
Our experiments are conducted on three public RGB-D datasets: the RHD dataset {{cite:6e172e3}}, the Stereo dataset {{cite:c61ca52}}, and the Dexter-Object dataset {{cite:036c06d}}, as shown in Table {{table:d71480d2-fef3-4a47-a764-0e0f371c1bd8}} .
The RHD dataset is created synthetically and contains 41,238 training and 2,728 test images, with a resolution of {{formula:ccbcd137-398e-4086-b673-e344bd89de8e}} . Each pair of RGB and depth images contains 3D annotations for 21 hand joints, and intrinsic camera parameters. The RHD dataset is built from 20 different subjects performing 39 actions. The training set has 16 subjects performing 31 actions, while the test set has 4 subjects performing 8 actions. The dataset contains diverse backgrounds sampled from 1,231 Flickr images.
The Stereo {{cite:c61ca52}} dataset is a real RGB-D dataset, which has 18,000 pairs of RGB and depth images with a resolution of {{formula:01abc23e-be28-47bc-b991-91ff6b95613a}}  pixels. Each pair is fully annotated with 21 joints. The dataset contains six different backgrounds with respect to different difficulties (textured/textureless, dynamic/static, near/far, hightlights/no-highlights). For each background, there are two sequences, each containing 1,500 image pairs. The dataset is manually annotated. In our experiments, we follow the evaluation protocol of {{cite:6e172e3}}, , we train on 10 sequences (15,000 images) and test on the remaining 2 sequences (3,000 images).
The Dexter-Object {{cite:036c06d}} dataset contains 3,111 images of two subjects performing manipulations with a cuboid. The dataset provides RGB and depth images, but only fingertips are annotated. The RGB images have a resolution of {{formula:88a3133a-2e7f-4c55-b048-687a8433bffc}}  pixels. Due to the incomplete hand annotation, we use this dataset for cross-dataset generalization.


System spans up to 3 km with concurrent end devices

-LoRa has good resistance to interference

-LoRa covers up to 3 km in suburban areas

A Study of LoRa: Long Range & Low Power Networks for the Internet of Things

Abstract: LoRa is a long-range, low-power, low-bitrate, wireless telecommunications system, promoted as an infrastructure solution for the Internet of Things: end-devices use LoRa across a single wireless hop to communicate to gateway(s), connected to the Internet and which act as transparent bridges and relay messages between these end-devices and a central network server. This paper provides an overview of LoRa and an in-depth analysis of its functional components. The physical and data link layer performance is evaluated by field tests and simulations. Based on the analysis and evaluations, some possible solutions for performance enhancements are proposed.

Analysis of Latency and MAC-Layer Performance for Class A LoRaWAN

Abstract: We propose analytical models that allow us to investigate the performance of long range wide area network (LoRaWAN) uplink in terms of latency, collision rate, and throughput under the constraints of the regulatory duty cycling, when assuming exponential inter-arrival times. Our models take into account sub-band selection and the case of sub-band combining. Our numerical evaluations consider specifically the European ISM band, but the analysis is applicable to any coherent band. Protocol simulations are used to validate the proposed models. We find that sub-band selection and combining have a large effect on the quality of service (QoS) experienced in an LoRaWAN cell for a given load. The proposed models allow for the optimization of resource allocation within a cell given a set of QoS requirements and a traffic model.

Understanding the Limits of LoRaWAN

Abstract: Low-power wide area networking technology offers long-range communication, which enables new types of services. Several solutions exist; LoRaWAN is arguably the most adopted. It promises ubiquitous connectivity in outdoor IoT applications, while keeping network structures and management simple. This technology has received a lot of attention in recent months from network operators and solution providers. However, the technology has limitations that need to be clearly understood to avoid inflated expectations and disillusionment. This article provides an impartial and fair overview of the capabilities and limitations of LoRaWAN. We discuss those in the context of use cases, and list open research and development questions.

Summary of LPWA-related research activities

Challenges and future research directions

There have been extensive efforts by the research community to optimize or evaluate LPWA operation. Table {{table:1a465156-7b85-452c-bfc0-06bfae6a52dc}}  summarizes these contributions. These works, however, have not addressed all issues associated with the LPWA network systems. In this section, we discuss current LPWA challenges and present directions for future research in this field.


Weakly Supervised Medical Diagnosis and Localization from Multiple Resolutions

Abstract: Diagnostic imaging often requires the simultaneous identification of a multitude of findings of varied size and appearance. Beyond global indication of said findings, the prediction and display of localization information improves trust in and understanding of results when augmenting clinical workflow. Medical training data rarely includes more than global image-level labels as segmentations are time-consuming and expensive to collect. We introduce an approach to managing these practical constraints by applying a novel architecture which learns at multiple resolutions while generating saliency maps with weak supervision. Further, we parameterize the Log-Sum-Exp pooling function with a learnable lower-bounded adaptation (LSE-LBA) to build in a sharpness prior and better handle localizing abnormalities of different sizes using only image-level labels. Applying this approach to interpreting chest x-rays, we set the state of the art on 9 abnormalities in the NIH's CXR14 dataset while generating saliency maps with the highest resolution to date.

ChestNet: A Deep Neural Network for Classification of Thoracic Diseases on Chest Radiography

Abstract: Computer-aided techniques may lead to more accurate and more acces-sible diagnosis of thorax diseases on chest radiography. Despite the success of deep learning-based solutions, this task remains a major challenge in smart healthcare, since it is intrinsically a weakly supervised learning problem. In this paper, we incorporate the attention mechanism into a deep convolutional neural network, and thus propose the ChestNet model to address effective diagnosis of thorax diseases on chest radiography. This model consists of two branches: a classification branch serves as a uniform feature extraction-classification network to free users from troublesome handcrafted feature extraction, and an attention branch exploits the correlation between class labels and the locations of patholog-ical abnormalities and allows the model to concentrate adaptively on the patholog-ically abnormal regions. We evaluated our model against three state-of-the-art deep learning models on the Chest X-ray 14 dataset using the official patient-wise split. The results indicate that our model outperforms other methods, which use no extra training data, in diagnosing 14 thorax diseases on chest radiography.

Thoracic Disease Identification and Localization with Limited Supervision

Abstract: Accurate identification and localization of abnormalities from radiology images play an integral part in clinical diagnosis and treatment planning. Building a highly accurate prediction model for these tasks usually requires a large number of images manually annotated with labels and finding sites of abnormalities. In reality, however, such annotated data are expensive to acquire, especially the ones with location annotations. We need methods that can work well with only a small amount of location annotations. To address this challenge, we present a unified approach that simultaneously performs disease identification and localization through the same underlying model for all images. We demonstrate that our approach can effectively leverage both class information as well as limited location annotation, and significantly outperforms the comparative reference baseline in both classification and localization tasks.

ImageGCN: Multi-Relational Image Graph Convolutional Networks for Disease Identification With Chest X-Rays

Abstract: Image representation is a fundamental task in computer vision. However, most of the existing approaches for image representation ignore the relations between images and consider each input image independently. Intuitively, relations between images can help to understand the images and maintain model consistency over related images, leading to better explainability. In this paper, we consider modeling the image-level relations to generate more informative image representations, and propose ImageGCN, an end-to-end graph convolutional network framework for inductive multi-relational image modeling. We apply ImageGCN to chest X-ray images where rich relational information is available for disease identification. Unlike previous image representation models, ImageGCN learns the representation of an image using both its original pixel features and its relationship with other images. Besides learning informative representations for images, ImageGCN can also be used for object detection in a weakly supervised manner. The experimental results on 3 open-source x-ray datasets, ChestX-ray14, CheXpert and MIMIC-CXR demonstrate that ImageGCN can outperform respective baselines in both disease identification and localization tasks and can achieve comparable and often better results than the state-of-the-art methods.

CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning

Abstract: We develop an algorithm that can detect pneumonia from chest X-rays at a level exceeding practicing radiologists. Our algorithm, CheXNet, is a 121-layer convolutional neural network trained on ChestX-ray14, currently the largest publicly available chest X-ray dataset, containing over 100,000 frontal-view X-ray images with 14 diseases. Four practicing academic radiologists annotate a test set, on which we compare the performance of CheXNet to that of radiologists. We find that CheXNet exceeds average radiologist performance on the F1 metric. We extend CheXNet to detect all 14 diseases in ChestX-ray14 and achieve state of the art results on all 14 diseases.

Reported performance of previous works on CXR14 dataset. Each work differs in augmentation schemes and some even in the usage of the dataset. We choose CheXNet {{cite:8b4ac8f}} as the baseline model for adding context modules.

As shown in Table {{table:b9c134a3-4b43-413b-b30a-d9e815922c9b}} , previous works on CXR14 dataset vary in loss, input size, etc.
We use CheXNet {{cite:8b4ac8f}} implementationhttps://github.com/jrzech/reproduce-chexnet to conduct context module comparisons, and varied with the backbone architecture and the input size to find if context modules work in various settings.
We use BCE loss with the SGD optimizer with momentum 0.9 and weight decay 0.0001.
Although ChexNet uses the input size of 224, we use the input size of 448 as it shows a better result than 224 with DenseNet121.
More training details can be found in the supplementary material.


Moment-to-moment Engagement Prediction through the Eyes of the Observer: PUBG Streaming on Twitch

Abstract: Is it possible to predict moment-to-moment gameplay engagement based solely on game telemetry? Can we reveal engaging moments of gameplay by observing the way the viewers of the game behave? To address these questions in this paper, we reframe the way gameplay engagement is defined and we view it, instead, through the eyes of a game's live audience. We build prediction models for viewers' engagement based on data collected from the popular battle royale game PlayerUnknown's Battlegrounds as obtained from the Twitch streaming service. In particular, we collect viewers' chat logs and in-game telemetry data from several hundred matches of five popular streamers (containing over 100,000 game events) and machine learn the mapping between gameplay and viewer chat frequency during play, using small neural network architectures. Our key findings showcase that engagement models trained solely on 40 gameplay features can reach accuracies of up to 80% on average and 84% at best. Our models are scalable and generalisable as they perform equally well within- and across-streamers, as well as across streamer play styles.

Real-time deep dynamic characters

Abstract: We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit spacetime coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.

Summary of surveyed studies associated with the four core phases of the affective game loop—Elicitation, Sensing, Detection, and Adaptation. The studies are placed under each of the four phases based on their primary focus within the affective loop. They are also placed under one of the main categories of each phase based on their research emphasis.

Table {{table:7f5bce06-a839-4b9c-acf5-adeab9c3c581}}  shows the outcome of this survey complemented with seminal works over the last 20 years of affective game computing. The table is organised based on the core aspects of the affective game loop—elicitation, sensing, detection, and adaptation. While many of the presented papers utilise multiple—if not all—aspects of the affective loop, we have selected papers that focus on a specific phase of the loop with the corresponding high-level categories defined in earlier sections of the paper.

In this section we discuss overall observations regarding the papers identified under each one of the four main phases of the affective loop (Table {{table:7f5bce06-a839-4b9c-acf5-adeab9c3c581}} .) Elicitation encompasses papers that focus on the emotional impact of games and their measurement. We divided these papers based on the game facet they consider. While many studies seem to put an emphasis on the game context, most of them focus on specific genres and platforms. In particular, the horror genre {{cite:02760de}}, {{cite:28f453a}}, {{cite:78ca010}} provides a popular research test-bed, possibly due to the visceral emotions that emerge during consuming horror media. VR and AR games are also popular for studying the impact of gameplay context on player experience {{cite:d9a2a9e}}, {{cite:c294513}}, {{cite:ddc093f}}. Even though there are some studies dedicated to the ways the action space of the game affects the player experience {{cite:aad5ed0}}, {{cite:8c3bd8c}}, there are no studies found that focus on the impact of the number of players, observability, stochasticity, and time granularity on player affect. When it comes to AI agents, there is surprisingly little research dedicated on how these agents can be used for emotion elicitation despite the field of believable non-player characters {{cite:60c8a5f}}, {{cite:a1a5da3}} being very active. Unsurprisingly, most studies with a primary focus on affect elicitation are investigating the impact of the game content and the game environment {{cite:5596aa3}}, virtual objects {{cite:524389b}}, sounds {{cite:c306b5c}}, game events {{cite:8c3bd8c}} or multiple facets of content {{cite:43fb0aa}} on affect elicitation. Summing it up, it appears that there are two major research gaps identified in game affect elicitation. First, there is lack of fundamental research into how components of the game context affect the player experience. Second, there is an untapped opportunity to utilise the research that has gone into creating more emotive and human-like agents for emotion elicitation in affective game computing studies.

Finally, we take a look at studies mostly linked to the phase of adaptation (bottom row of Table {{table:7f5bce06-a839-4b9c-acf5-adeab9c3c581}} ). One could argue that most commercial games that use some type of player-based adaptation employ a form of the affective loop. Similarly in academia, the research area of game affect adaptation is getting some serious traction in recent years. As Table {{table:7f5bce06-a839-4b9c-acf5-adeab9c3c581}}  shows, there are plenty of research projects that focus on affect adaptation and intelligent interaction based on sensing and emotion detection.
Unsurprisingly, similarly to studies focusing on elicitation, most studies involving game affect adaptation focus on the game content instead of interactive agents {{cite:64f50ce}}. While the holistic approach of experience-driven procedural content generation allows for more in-game adaptation opportunities {{cite:633b939}}, the lack of emotive agents in affect adaptation reveals a notable gap in the research field. Despite the successful affect-adaptive methods showcased in the literature, the widespread adoption of these techniques in commercial-standard games is still in its infancy. One of the biggest obstacles in this regard appears to be the field's reliance on often intrusive biosensors. While most physiological sensors can provide valuable multimodal data, such data are often very hard to reliably capture in the wild. We expect that future research avenues will investigate methods that would allow rapid deployment in games via camera-based technology or via the use of user-agnostic models—e.g. learning from in-game footage pixels or via privileged information {{cite:0185d84}}, {{cite:5c652dd}}.


Adaptive Control of Exoskeleton Robots for Periodic Assistive Behaviours Based on EMG Feedback Minimisation

Abstract: In this paper we propose an exoskeleton control method for adaptive learning of assistive joint torque profiles in periodic tasks. We use human muscle activity as feedback to adapt the assistive joint torque behaviour in a way that the muscle activity is minimised. The user can then relax while the exoskeleton takes over the task execution. If the task is altered and the existing assistive behaviour becomes inadequate, the exoskeleton gradually adapts to the new task execution so that the increased muscle activity caused by the new desired task can be reduced. The advantage of the proposed method is that it does not require biomechanical or dynamical models. Our proposed learning system uses Dynamical Movement Primitives (DMPs) as a trajectory generator and parameters of DMPs are modulated using Locally Weighted Regression. Then, the learning system is combined with adaptive oscillators that determine the phase and frequency of motion according to measured Electromyography (EMG) signals. We tested the method with real robot experiments where subjects wearing an elbow exoskeleton had to move an object of an unknown mass according to a predefined reference motion. We further evaluated the proposed approach on a whole-arm exoskeleton to show that it is able to adaptively derive assistive torques even for multiple-joint motion.

Geometry-aware Dynamic Movement Primitives

Abstract: In many robot control problems, factors such as stiffness and damping matrices and manipulability ellipsoids are naturally represented as symmetric positive definite (SPD) matrices, which capture the specific geometric characteristics of those factors. Typical learned skill models such as dynamic movement primitives (DMPs) can not, however, be directly employed with quantities expressed as SPD matrices as they are limited to data in Euclidean space. In this paper, we propose a novel and mathematically principled framework that uses Riemannian metrics to reformulate DMPs such that the resulting formulation can operate with SPD data in the SPD manifold. Evaluation of the approach demonstrates that beneficial properties of DMPs such as change of the goal during operation apply also to the proposed formulation.

Merging Position and Orientation Motion Primitives

Abstract: In this paper, we focus on generating complex robotic trajectories by merging sequential motion primitives. A robotic trajectory is a time series of positions and orientations ending at a desired target. Hence, we first discuss the generation of converging pose trajectories via dynamical systems, providing a rigorous stability analysis. Then, we present approaches to merge motion primitives which represent both the position and the orientation part of the motion. Developed approaches preserve the shape of each learned movement and allow for continuous transitions among succeeding motion primitives. Presented methodologies are theoretically described and experimentally evaluated, showing that it is possible to generate a smooth pose trajectory out of multiple motion primitives.

Open-source implementations of dmp-based approaches that we have released to the community. The source code for each appraoch is available at https://gitlab.com/dmp-codes-collection

An important and desired feature of any motion primitive representation is the possibility to combine basic movements to obtain more complex behaviors {{cite:d7cf07f}}. We review here three prominent approaches developed to smoothly join a sequence of dmp. In this tutorial, we name the approach by {{cite:937cbc4}} as velocity threshold, that in {{cite:7c46c19}} as target crossing, and that in {{cite:c4654b3}}, {{cite:5bb7bfc}} as basis functions overlay. Some of the presented approaches modify the dmp formulations in Section [subsec:discrete:classical]subsec:discrete:classical and  REF . The main differences are highlighted with green text. The 3 approaches have been implemented in Matlab for both position (Section REF ) and orientation (Section REF ) dmp. The source code is included in our public repository (see Table {{table:d9ef2fb6-571a-4976-b8b2-c4c3562d3fe2}} ). Results on synthetic data are shown in Figures [fig:Joiningzerovelocity]fig:Joiningzerovelocity to {{figure:1aab4e4f-2971-47c8-89d8-d1ea6a0b85fe}} .


The pseudo-compartment method for coupling PDE and compartment-based models of diffusion

Abstract: Spatial reaction-diffusion models have been employed to describe many emergent phenomena in biological systems. The modelling technique most commonly adopted in the literature implements systems of partial differential equations (PDEs), which assumes there are sufficient densities of particles that a continuum approximation is valid. However, due to recent advances in computational power, the simulation, and therefore postulation, of computationally intensive individual-based models has become a popular way to investigate the effects of noise in reaction-diffusion systems in which regions of low copy numbers exist. The stochastic models with which we shall be concerned in this manuscript are referred to as `compartment-based'. These models are characterised by a discretisation of the computational domain into a grid/lattice of `compartments'. Within each compartment particles are assumed to be well-mixed and are permitted to react with other particles within their compartment or to transfer between neighbouring compartments. We develop two hybrid algorithms in which a PDE is coupled to a compartment-based model. Rather than attempting to balance average fluxes, our algorithms answer a more fundamental question: `how are individual particles transported between the vastly different model descriptions?' First, we present an algorithm derived by carefully re-defining the continuous PDE concentration as a probability distribution. Whilst this first algorithm shows strong convergence to analytic solutions of test problems, it can be cumbersome to simulate. Our second algorithm is a simplified and more efficient implementation of the first, it is derived in the continuum limit over the PDE region alone. We test our hybrid methods for functionality and accuracy in a variety of different scenarios by comparing the averaged simulations to analytic solutions of PDEs for mean concentrations.

Hybrid method for simulating front propagation in reaction-diffusion systems

Abstract: We study the propagation of pulled fronts in the $A <-> \leftrightarrow A+A$ microscopic reaction-diffusion process using Monte Carlo (MC) simulations. In the mean field approximation the process is described by the deterministic Fisher-Kolmogorov-Petrovsky-Piscounov (FKPP) equation. In particular we concentrate on the corrections to the deterministic behavior due to the number of particles per site $\Omega$. By means of a new hybrid simulation scheme, we manage to reach large macroscopic values of $\Omega$ which allows us to show the importance in the dynamics of microscopic pulled fronts of the interplay of microscopic fluctuations and their macroscopic relaxation.

Hybrid approaches for multiple-species stochastic reaction-diffusion models

Abstract: Reaction-diffusion models are used to describe systems in fields as diverse as physics, chemistry, ecology and biology. The fundamental quantities in such models are individual entities such as atoms and molecules, bacteria, cells or animals, which move and/or react in a stochastic manner. If the number of entities is large, accounting for each individual is inefficient, and often partial differential equation (PDE) models are used in which the stochastic behaviour of individuals is replaced by a description of the averaged, or mean behaviour of the system. In some situations the number of individuals is large in certain regions and small in others. In such cases, a stochastic model may be inefficient in one region, and a PDE model inaccurate in another. To overcome this problem, we develop a scheme which couples a stochastic reaction-diffusion system in one part of the domain with its mean field analogue, i.e. a discretised PDE model, in the other part of the domain. The interface in between the two domains occupies exactly one lattice site and is chosen such that the mean field description is still accurate there. This way errors due to the flux between the domains are small. Our scheme can account for multiple dynamic interfaces separating multiple stochastic and deterministic domains, and the coupling between the domains conserves the total number of particles. The method preserves stochastic features such as extinction not observable in the mean field description, and is significantly faster to simulate on a computer than the pure stochastic model.

A hybrid algorithm for coupling PDE and compartment-based dynamics

Abstract: Stochastic simulation methods can be applied successfully to model exact spatio-temporally resolved reaction-diffusion systems. However, in many cases, these methods can quickly become extremely computationally intensive with increasing particle numbers. An alternative description of many of these systems can be derived in the diffusive limit as a deterministic, continuum system of partial differential equations. Although the numerical solution of such partial differential equations is, in general, much more efficient than the full stochastic simulation, the deterministic continuum description is generally not valid when copy numbers are low and stochastic effects dominate. Therefore, to take advantage of the benefits of both of these types of models, each of which may be appropriate in different parts of a spatial domain, we have developed an algorithm that can be used to couple these two types of model together. This hybrid coupling algorithm uses an overlap region between the two modelling regimes. By coupling fluxes at one end of the interface and using a concentration-matching condition at the other end, we ensure that mass is appropriately transferred between PDE- and compartment-based regimes. Our methodology gives notable reductions in simulation time in comparison with using a fully stochastic model, whilst maintaining the important stochastic features of the system and providing detail in appropriate areas of the domain. We test our hybrid methodology robustly by applying it to several biologically motivated problems including diffusion and morphogen gradient formation. Our analysis shows that the resulting error is small, unbiased and does not grow over time.

A summary of the macro-meso hybrid papers that will be covered in this section. The “type” column gives a brief description of the type of coupling used to join the two regimes. “Spatially-coupled” means that the domain is split into two distinct regions within which different paradigms are used. “Adaptive” refers to whether an interface is able to move, while “overlap” indicates if an overlap region is investigated. “Operator splitting” indicates where reaction and diffusion are modelled in different ways, rather than dividing space, and “propensity-based spatial splitting” is where the propensity functions are split based on their value. The “system modelled” column describes the application for which these models can be used. The “Method or application?” column details whether the paper introduces a novel hybrid method or is based around the application of a currently existing method to a particular problem. All of the macro-meso hybrid papers present novel methods rather than applications of pre-existing methods to real-world systems.

In this section, we will first introduce the broad concept, and then review specific examples of models which couple macroscopic dynamics to mesoscopic dynamics, which we will refer to as “macro-meso” hybrid models. We list and describe the macro-meso hybrid methods covered in this section in Table {{table:50f93b68-0eed-4366-aa46-201c6ad39b56}} . We begin by giving an illustrative example of a macro-meso hybrid model, the pseudo-compartment method (PCM) {{cite:28eb1e8}} and present pseudocode for its implementation. We then summarise several other existing macro-meso hybrid methods and present schematics (where appropriate) to aid the reader's understanding.


Introduce context optimization with learnable text prompts for VLM transfer learning.

Propose subspace text prompt tuning to mitigate overfitting in VLM transfer learning.

Propose prompt distribution learning that captures the distribution of diverse text prompts.

Present a prompt-aligned gradient technique for preventing knowledge forgetting.

Employ counterfactual generation and contrastive learning for text prompt tuning.

Introduce optimal transport to learn multiple comprehensive text prompts.

Introduce positive and negative text prompt learning for multi-label classification.

Introduce a double-grained prompt tuning technique for multi-label classification

Propose to fine-tune VLMs on multiple downstream tasks simultaneously.

Propose a language-guided fine-tuning technique for dense visual recognition tasks.

Employ large-scale language models to generate customized prompts for VLMs.

Propose unsupervised prompt learning with self-training for VLM transfer learning.

Propose test-time prompt tuning that learns adaptive prompts on the fly.

Investigate the efficacy of visual prompt tuning for VLM transfer learning.

Propose a unified prompt tuning that jointly optimizes text and image prompts.

Incorporate multi-task knowledge into text and image prompt tuning.

Propose multi-modal prompt tuning with a mutual promotion strategy.

Propose to build a training-free adapter with the embeddings of few labelled images.

Introduce a self-supervised adapter by performing self-supervised learning on images.

Propose a training-free name-only transfer learning paradigm with curated support sets.

Leverage the label distribution priors for adapting pre-trained VLMs.

Introduce visual-guided attention that semantically aligns text and image features.

Propose parameter-free attention for the communication between visual and textual features.

Propose to extract dense features by modifying the image encoder architecture.

Learning to Prompt for Vision-Language Models

Abstract: Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming -- one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt's context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.

Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models

Abstract: Pretrained vision-language models (VLMs) such as CLIP have shown impressive generalization capability in downstream vision tasks with appropriate text prompts. Instead of designing prompts manually, Context Optimization (CoOp) has been recently proposed to learn continuous prompts using taskspecific training data. Despite the performance improvements on downstream tasks, several studies have reported that CoOp suffers from the overfitting issue in two aspects: (i) the test accuracy on base classes first improves and then worsens during training;(ii) the test accuracy on novel classes keeps decreasing. However, none of the existing studies can understand and mitigate such overfitting problems. In this study, we first explore the cause of overfitting by analyzing the gradient flow. Comparative experiments reveal that CoOp favors generalizable and spurious features in the early and later training stages, respectively, leading to the non-overfitting and overfitting phenomena. Given those observations, we propose Subspace Prompt Tuning (SubPT) to project the gradients in back-propagation onto the low-rank subspace spanned by the early-stage gradient flow eigenvectors during the entire training process and successfully eliminate the overfitting problem. In addition, we equip CoOp with a Novel Feature Learner (NFL) to enhance the generalization ability of the learned prompts onto novel categories beyond the training set, needless of image training data. Extensive experiments on 11 classification datasets demonstrate that SubPT+NFL consistently boost the performance of CoOp and outperform the state-of-the-art CoCoOp approach. Experiments on more challenging vision downstream tasks, including open-vocabulary object detection and zero-shot semantic segmentation, also verify the effectiveness of the proposed method. Codes can be found at https://tinyurl.com/mpe64f89.

Prompt Distribution Learning

Abstract: We present prompt distribution learning for effectively adapting a pre-trained vision-language model to address downstream recognition tasks. Our method not only learns low-bias prompts from a few samples but also captures the distribution of diverse prompts to handle the varying visual representations. In this way, we provide high-quality task-related content for facilitating recognition. This prompt distribution learning is realized by an efficient approach that learns the output embeddings of prompts instead of the input embeddings. Thus, we can employ a Gaussian distribution to model them effectively and derive a surrogate loss for efficient training. Extensive experiments on 12 datasets demonstrate that our method consistently and significantly outperforms existing methods. For example, with 1 sample per category, it relatively improves the average result by 9.1% compared to human-crafted prompts.

Prompt-aligned Gradient for Prompt Tuning

Abstract: Thanks to the large pre-trained vision-language models (VLMs) like CLIP, we can craft a zero-shot classifier by "prompt", e.g., the confidence score of an image being "[CLASS]" can be obtained by using the VLM provided similarity measure between the image and the prompt sentence "a photo of a [CLASS]". Therefore, prompt shows a great potential for fast adaptation of VLMs to downstream tasks if we fine-tune the prompt-based similarity measure. However, we find a common failure that improper fine-tuning may not only undermine the prompt's inherent prediction for the task-related classes, but also for other classes in the VLM vocabulary. Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific to prompt. We present Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from forgetting the the general knowledge learned from VLMs. In particular, ProGrad only updates the prompt whose gradient is aligned (or non-conflicting) to the "general direction", which is represented as the gradient of the KL loss of the pre-defined prompt prediction. Extensive experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods. Codes are available at https://github.com/BeierZhu/Prompt-align.

CPL: Counterfactual Prompt Learning for Vision and Language Models

Abstract: Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP. However, existing prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen concepts.Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel Counterfactual Prompt Learning (CPL) method for vision and language models, which simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework.Particularly, CPL constructs counterfactual by identifying minimal non-spurious feature change between semantically-similar positive and negative samples that causes concept change, and learns more generalizable prompt representation from both factual and counterfactual examples via contrastive learning. Extensive experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks than previous prompt tuning methods on CLIP. On image classification, we achieve 3.55% average relative improvement on unseen classes across seven datasets; on image-text retrieval and visual question answering, we gain up to 4.09% and 25.08% relative improvements across three few-shot scenarios on unseen test sets respectively.

PLOT: Prompt Learning with Optimal Transport for Vision-Language Models

Abstract: With the increasing attention to large vision-language models such as CLIP, there has been a significant amount of effort dedicated to building efficient prompts. Unlike conventional methods of only learning one single prompt, we propose to learn multiple comprehensive prompts to describe diverse characteristics of categories such as intrinsic attributes or extrinsic contexts. However, directly matching each prompt to the same visual feature is problematic, as it pushes the prompts to converge to one point. To solve this problem, we propose to apply optimal transport to match the vision and text modalities. Specifically, we first model images and the categories with visual and textual feature sets. Then, we apply a two-stage optimization strategy to learn the prompts. In the inner loop, we optimize the optimal transport distance to align visual features and prompts by the Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distance from the supervised data. Extensive experiments are conducted on the few-shot recognition task and the improvement demonstrates the superiority of our method. The code is available at https://github.com/CHENGY12/PLOT.

DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations

Abstract: Solving multi-label recognition (MLR) for images in the low-label regime is a challenging task with many real-world applications. Recent work learns an alignment between textual and visual spaces to compensate for insufficient image labels, but loses accuracy because of the limited amount of available MLR annotations. In this work, we utilize the strong alignment of textual and visual features pretrained with millions of auxiliary image-text pairs and propose Dual Context Optimization (DualCoOp) as a unified framework for partial-label MLR and zero-shot MLR. DualCoOp encodes positive and negative contexts with class names as part of the linguistic input (i.e. prompts). Since DualCoOp only introduces a very light learnable overhead upon the pretrained vision-language framework, it can quickly adapt to multi-label recognition tasks that have limited annotations and even unseen classes. Experiments on standard multi-label recognition benchmarks across two challenging low-label settings demonstrate the advantages of our approach over state-of-the-art methods.

Texts as Images in Prompt Tuning for Multi-Label Image Recognition

Abstract: Prompt tuning has been employed as an efficient way to adapt large vision-language pre-trained models (e.g. CLIP) to various downstream tasks in data-limited or label-limited settings. Nonetheless, visual data (e.g., images) is by default prerequisite for learning prompts in existing methods. In this work, we advocate that the effectiveness of image-text contrastive learning in aligning the two modalities (for training CLIP) further makes it feasible to treat texts as images for prompt tuning and introduce TaI prompting. In contrast to the visual data, text descriptions are easy to collect, and their class labels can be directly derived. Particularly, we apply TaI prompting to multi-label image recognition, where sentences in the wild serve as alternatives to images for prompt tuning. Moreover, with TaI, double-grained prompt tuning (TaI-DPT) is further presented to extract both coarse-grained and fine-grained embeddings for enhancing the multi-label recognition performance. Experimental results show that our proposed TaI-DPT outperforms zero-shot CLIP by a large margin on multiple benchmarks, e.g., MS-COCO, VOC2007, and NUS-WIDE, while it can be combined with existing methods of prompting from images to improve recognition performance further. Code is released at https://github.com/guozix/TaI-DPT.

Prompt Tuning with Soft Context Sharing for Vision-Language Models

Abstract: Vision-language models have recently shown great potential on many computer vision tasks. Meanwhile, prior work demonstrates prompt tuning designed for vision-language models could acquire superior performance on few-shot image recognition compared to linear probe, a strong baseline. In real-world applications, many few-shot tasks are correlated, particularly in a specialized area. However, such information is ignored by previous work. Inspired by the fact that modeling task relationships by multi-task learning can usually boost performance, we propose a novel method SoftCPT (Soft Context Sharing for Prompt Tuning) to fine-tune pre-trained vision-language models on multiple target few-shot tasks, simultaneously. Specifically, we design a task-shared meta network to generate prompt vector for each task using pre-defined task name together with a learnable meta prompt as input. As such, the prompt vectors of all tasks will be shared in a soft manner. The parameters of this shared meta network as well as the meta prompt vector are tuned on the joint training set of all target tasks. Extensive experiments on three multi-task few-shot datasets show that SoftCPT outperforms the representative single-task prompt tuning method CoOp [78] by a large margin, implying the effectiveness of multi-task learning in vision-language prompt tuning. The source code and data will be made publicly available.

DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

Abstract: Recent progress has shown that large-scale pre-training using contrastive image-text pairs can be a promising alternative for high-quality visual representation learning from natural language supervision. Benefiting from a broader source of supervision, this new paradigm exhibits impressive transferability to downstream classification tasks and datasets. However, the problem of transferring the knowledge learned from image-text pairs to more complex dense prediction tasks has barely been visited. In this work, we present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. By further using the contextual information from the image to prompt the language model, we are able to facilitate our model to better exploit the pre-trained knowledge. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones including both CLIP models and ImageNet pre-trained models. Extensive experiments demonstrate the superior performance of our methods on semantic segmentation, object detection, and instance segmentation tasks. Code is available at https://github.com/raoyongming/DenseCLIP

What does a platypus look like? Generating customized prompts for zero-shot image classification

Abstract: Open-vocabulary models are a promising new paradigm for image classification. Unlike traditional classification models, open-vocabulary models classify among any arbitrary set of categories specified with natural language during inference. This natural language, called "prompts", typically consists of a set of hand-written templates (e.g., "a photo of a {}") which are completed with each of the category names. This work introduces a simple method to generate higher accuracy prompts, without relying on any explicit knowledge of the task domain and with far fewer hand-constructed sentences. To achieve this, we combine open-vocabulary models with large language models (LLMs) to create Customized Prompts via Language models (CuPL, pronounced "couple"). In particular, we leverage the knowledge contained in LLMs in order to generate many descriptive sentences that contain important discriminating characteristics of the image categories. This allows the model to place a greater importance on these regions in the image when making predictions. We find that this straightforward and general approach improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet. Finally, this simple baseline requires no additional training and remains completely zero-shot. Code available at https://github.com/sarahpratt/CuPL.

Unsupervised Prompt Learning for Vision-Language Models

Abstract: Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explore a different scenario, in which the labels of the target datasets are unprovided, and we present an unsupervised prompt learning (UPL) approach to avoid prompt engineering while simultaneously improving transfer performance of CLIP-like vision-language models. As far as we know, UPL is the first work to introduce unsupervised learning into prompt learning. Experimentally, our UPL outperforms original CLIP with prompt engineering on ImageNet as well as other 10 datasets. An enhanced version of UPL is even competitive with the 8-shot CoOp and the 8-shot TIP-Adapter on most datasets. Code and models are available at https://github.com/tonyhuang2022/UPL.

Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models

Abstract: Pre-trained vision-language models (e.g., CLIP) have shown promising zero-shot generalization in many downstream tasks with properly designed text prompts. Instead of relying on hand-engineered prompts, recent works learn prompts using the training data from downstream tasks. While effective, training on domain-specific data reduces a model's generalization capability to unseen new domains. In this work, we propose test-time prompt tuning (TPT), a method that can learn adaptive prompts on the fly with a single test sample. For image classification, TPT optimizes the prompt by minimizing the entropy with confidence selection so that the model has consistent predictions across different augmented views of each test sample. In evaluating generalization to natural distribution shifts, TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average, surpassing previous prompt tuning approaches that require additional task-specific training data. In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data. Project page: https://azshue.github.io/TPT.

Exploring Visual Prompts for Adapting Large-Scale Models

Abstract: We investigate the efficacy of visual prompting to adapt large-scale models in vision. Following the recent approach from prompt tuning and adversarial reprogramming, we learn a single image perturbation such that a frozen model prompted with this perturbation performs a new task. Through comprehensive experiments, we demonstrate that visual prompting is particularly effective for CLIP and robust to distribution shift, achieving performance competitive with standard linear probes. We further analyze properties of the downstream dataset, prompt design, and output transformation in regard to adaptation performance. The surprising effectiveness of visual prompting provides a new perspective on adapting pre-trained models in vision. Code is available at http://hjbahng.github.io/visual_prompting .

Unified Vision and Language Prompt Learning

Abstract: Prompt tuning, a parameter- and data-efficient transfer learning paradigm that tunes only a small number of parameters in a model's input space, has become a trend in the vision community since the emergence of large vision-language models like CLIP. We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that none of the unimodal prompt tuning methods performs consistently well: text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities. Extensive experiments on over 11 vision datasets show that UPT achieves a better trade-off than the unimodal counterparts on few-shot learning benchmarks, as well as on domain generalization benchmarks. Code and models will be released to facilitate future research.

Multitask Vision-Language Prompt Tuning

Abstract: Prompt Tuning, conditioning on task-specific learned prompt vectors, has emerged as a data-efficient and parameter-efficient method for adapting large pretrained vision-language models to multiple downstream tasks. However, existing approaches usually consider learning prompt vectors for each task independently from scratch, thereby failing to exploit the rich shareable knowledge across different vision-language tasks. In this paper, we propose multitask vision-language prompt tuning (MVLPT), which incorporates cross-task knowledge into prompt tuning for vision-language models. Specifically, (i) we demonstrate the effectiveness of learning a single transferable prompt from multiple source tasks to initialize the prompt for each target task; (ii) we show many target tasks can benefit each other from sharing prompt vectors and thus can be jointly learned via multitask prompt tuning. We benchmark the proposed MVLPT using three representative prompt tuning methods, namely text prompt tuning, visual prompt tuning, and the unified vision-language prompt tuning. Results in 20 vision tasks demonstrate that the proposed approach outperforms all single-task baseline prompt tuning methods, setting the new state-of-the-art on the few-shot ELEVATER benchmarks and cross-task generalization benchmarks. To understand where the cross-task knowledge is most effective, we also conduct a large-scale study on task transferability with 20 vision tasks in 400 combinations for each prompt tuning method. It shows that the most performant MVLPT for each prompt tuning method prefers different task combinations and many tasks can benefit each other, depending on their visual similarity and label similarity. Code is available at https://github.com/sIncerass/MVLPT.

MaPLe: Multi-modal Prompt Learning

Abstract: Pre-trained vision-language (V-L) models such as CLIP have shown excellent generalization ability to downstream tasks. However, they are sensitive to the choice of input text prompts and require careful selection of prompt templates to perform well. Inspired by the Natural Language Processing (NLP) literature, recent CLIP adaptation approaches learn prompts as the textual inputs to fine-tune CLIP for downstream tasks. We note that using prompting to adapt representations in a single branch of CLIP (language or vision) is sub-optimal since it does not allow the flexibility to dynamically adjust both representation spaces on a downstream task. In this work, we propose Multi-modal Prompt Learning (MaPLe) for both vision and language branches to improve alignment between the vision and language representations. Our design promotes strong coupling between the vision-language prompts to ensure mutual synergy and discourages learning independent uni-modal solutions. Further, we learn separate prompts across different early stages to progressively model the stage-wise feature relationships to allow rich context learning. We evaluate the effectiveness of our approach on three representative tasks of generalization to novel classes, new target datasets and unseen domain shifts. Compared with the state-of-the-art method Co-CoOp, MaPLe exhibits favorable performance and achieves an absolute gain of 3.45% on novel classes and 2.72% on overall harmonic-mean, averaged over 11 diverse image recognition datasets. Our code and pre-trained models are available at https://github.com/muzairkhattak/multimodal-prompt-learning.

Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

Abstract: Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations by using large-scale contrastive image-text pairs. It shows impressive performance on zero-shot knowledge transfer to downstream tasks. To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification. However, such a process still needs extra training and computational resources. In this paper, we propose \textbf{T}raining-Free CL\textbf{IP}-\textbf{Adapter} (\textbf{Tip-Adapter}), which not only inherits CLIP's training-free advantage but also performs comparably or even better than CLIP-Adapter. Tip-Adapter does not require any back propagation for training the adapter, but creates the weights by a key-value cache model constructed from the few-shot training set. In this non-parametric manner, Tip-Adapter acquires well-performed adapter weights without any training, which is both efficient and effective. Moreover, the performance of Tip-Adapter can be further boosted by fine-tuning such properly initialized adapter for only a few epochs with super-fast convergence speed. We conduct extensive experiments of few-shot classification on ImageNet and other 10 datasets to demonstrate the superiority of proposed Tip-Adapter. The code will be released at \url{https://github.com/gaopengcuhk/Tip-Adapter}.

SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models

Abstract: Vision-language models such as CLIP are pretrained on large volumes of internet sourced image and text pairs, and have been shown to sometimes exhibit impressive zero- and low-shot image classification performance. However, due to their size, fine-tuning these models on new datasets can be prohibitively expensive, both in terms of the supervision and compute required. To combat this, a series of light-weight adaptation methods have been proposed to efficiently adapt such models when limited supervision is available. In this work, we show that while effective on internet-style datasets, even those remedies under-deliver on classification tasks with images that differ significantly from those commonly found online. To address this issue, we present a new approach called SVL-Adapter that combines the complementary strengths of both vision-language pretraining and self-supervised representation learning. We report an average classification accuracy improvement of 10% in the low-shot setting when compared to existing methods, on a set of challenging visual classification tasks. Further, we present a fully automatic way of selecting an important blending hyperparameter for our model that does not require any held-out labeled validation data. Code for our project is available here: https://github.com/omipan/svl_adapter.

SuS-X: Training-Free Name-Only Transfer of Vision-Language Models

Abstract: Contrastive Language-Image Pre-training (CLIP) has emerged as a simple yet effective way to train large-scale vision-language models. CLIP demonstrates impressive zero-shot classification and retrieval on diverse downstream tasks. However, to leverage its full potential, fine-tuning still appears to be necessary. Fine-tuning the entire CLIP model can be resource-intensive and unstable. Moreover, recent methods that aim to circumvent this need for fine-tuning still require access to images from the target distribution. In this paper, we pursue a different approach and explore the regime of training-free "name-only transfer" in which the only knowledge we possess about the downstream task comprises the names of downstream target categories. We propose a novel method, SuS-X, consisting of two key building blocks -- SuS and TIP-X, that requires neither intensive fine-tuning nor costly labelled data. SuS-X achieves state-of-the-art zero-shot classification results on 19 benchmark datasets. We further show the utility of TIP-X in the training-free few-shot setting, where we again achieve state-of-the-art results over strong training-free baselines. Code is available at https://github.com/vishaal27/SuS-X.

Improving Zero-Shot Models with Label Distribution Priors

Abstract: Labeling large image datasets with attributes such as facial age or object type is tedious and sometimes infeasible. Supervised machine learning methods provide a highly accurate solution, but require manual labels which are often unavailable. Zero-shot models (e.g., CLIP) do not require manual labels but are not as accurate as supervised ones, particularly when the attribute is numeric. We propose a new approach, CLIPPR (CLIP with Priors), which adapts zero-shot models for regression and classification on unlabelled datasets. Our method does not use any annotated images. Instead, we assume a prior over the label distribution in the dataset. We then train an adapter network on top of CLIP under two competing objectives: i) minimal change of predictions from the original CLIP model ii) minimal distance between predicted and prior distribution of labels. Additionally, we present a novel approach for selecting prompts for Vision & Language models using a distributional prior. Our method is effective and presents a significant improvement over the original model. We demonstrate an improvement of 28% in mean absolute error on the UTK age regression task. We also present promising results for classification benchmarks, improving the classification accuracy on the ImageNet dataset by 2.83%, without using any labels.

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts

Abstract: Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning. However, due to the semantic gap within datasets, CLIP's pre-trained image-text alignment becomes sub-optimal on downstream tasks, which severely harms its transferring performance. To better adapt the cross-modality embedding space, we propose to enhance CLIP via Visual-guided Texts, named VT-CLIP. Specifically, we guide textual features of different categories to adaptively explore informative regions on the image and aggregate visual features by attention mechanisms. In this way, the texts become visual-guided, namely, more semantically correlated with downstream images, which greatly benefits the category-wise matching process. In few-shot settings, we evaluate our VT-CLIP on 11 well-known classification datasets to demonstrate its effectiveness.

CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention

Abstract: Contrastive Language-Image Pre-training (CLIP) has been shown to learn visual representations with promising zero-shot performance. To further improve its downstream accuracy, existing works propose additional learnable modules upon CLIP and fine-tune them by few-shot training sets. However, the resulting extra training cost and data requirement severely hinder the efficiency for model deployment and knowledge transfer. In this paper, we introduce a free-lunch enhancement method, CALIP, to boost CLIP's zero-shot performance via a parameter-free attention module. Specifically, we guide visual and textual representations to interact with each other and explore cross-modal informative features via attention. As the pre-training has largely reduced the embedding distances between two modalities, we discard all learnable parameters in the attention and bidirectionally update the multi-modal features, enabling the whole process to be parameter-free and training-free. In this way, the images are blended with textual-aware signals and the text representations become visual-guided for better adaptive zero-shot alignment. We evaluate CALIP on various benchmarks of 14 datasets for both 2D image and 3D point cloud few-shot classification, showing consistent zero-shot performance improvement over CLIP. Based on that, we further insert a small number of linear layers in CALIP's attention module and verify our robustness under the few-shot settings, which also achieves leading performance compared to existing methods. Those extensive experiments demonstrate the superiority of our approach for efficient enhancement of CLIP. Code is available at https://github.com/ZiyuGuo99/CALIP.

Extract Free Dense Labels from CLIP

Abstract: Contrastive Language-Image Pre-training (CLIP) has made a remarkable breakthrough in open-vocabulary zero-shot image recognition. Many recent studies leverage the pre-trained CLIP models for image-level classification and manipulation. In this paper, we wish examine the intrinsic potential of CLIP for pixel-level dense prediction, specifically in semantic segmentation. To this end, with minimal modification, we show that MaskCLIP yields compelling segmentation results on open concepts across various datasets in the absence of annotations and fine-tuning. By adding pseudo labeling and self-training, MaskCLIP+ surpasses SOTA transductive zero-shot semantic segmentation methods by large margins, e.g., mIoUs of unseen classes on PASCAL VOC/PASCAL Context/COCO Stuff are improved from 35.6/20.7/30.3 to 86.1/66.7/54.7. We also test the robustness of MaskCLIP under input corruption and evaluate its capability in discriminating fine-grained objects and novel concepts. Our finding suggests that MaskCLIP can serve as a new reliable source of supervision for dense prediction tasks to achieve annotation-free segmentation. Source code is available at https://github.com/chongzhou96/MaskCLIP.

Summary of VLM transfer learning methods. TPT: text-prompt tuning; VPT: visual-prompt tuning; FA: feature adapter; CA: cross-attention; FT: fine-tuning; AM: architecture modification. [code] directs to code websites.

As shown in Table {{table:ee7717e7-f220-4a33-ac82-fe71c3020bbc}} , we broadly group existing VLM transfer methods into three categories including prompt tuning approaches, feature adapter approaches, and others.


Contact patterns in a high school: a comparison between data collected using wearable sensors, contact diaries and friendship surveys

Abstract: Given their importance in shaping social networks and determining how information or diseases propagate in a population, human interactions are the subject of many data collection efforts. To this aim, different methods are commonly used, from diaries and surveys to wearable sensors. These methods show advantages and limitations but are rarely compared in a given setting. As surveys targeting friendship relations might suffer less from memory biases than contact diaries, it is also interesting to explore how daily contact patterns compare with friendship relations and with online social links. Here we make progresses in these directions by leveraging data from a French high school: face-to-face contacts measured by two concurrent methods, sensors and diaries; self-reported friendship surveys; Facebook links. We compare the data sets and find that most short contacts are not reported in diaries while long contacts have larger reporting probability, with a general tendency to overestimate durations. Measured contacts corresponding to reported friendship can have durations of any length but all long contacts correspond to reported friendships. Online links not associated to reported friendships correspond to short face-to-face contacts, highlighting the different nature of reported friendships and online links. Diaries and surveys suffer from a low sampling rate, showing the higher acceptability of sensor-based platform. Despite the biases, we found that the overall structure of the contact network, i.e., the mixing patterns between classes, is correctly captured by both self-reported contacts and friendships networks. Overall, diaries and surveys tend to yield a correct picture of the structural organization of the contact network, albeit with much less links, and give access to a sort of backbone of the contact network corresponding to the strongest links in terms of cumulative durations.

Estimating Potential Infection Transmission Routes in Hospital Wards Using Wearable Proximity Sensors

Abstract: Contacts between patients, patients and health care workers (HCWs) and among HCWs represent one of the important routes of transmission of hospital-acquired infections (HAI). A detailed description and quantification of contacts in hospitals provides key information for HAIs epidemiology and for the design and validation of control measures. We used wearable sensors to detect close-range interactions ("contacts") between individuals in the geriatric unit of a university hospital. Contact events were measured with a spatial resolution of about 1.5 meters and a temporal resolution of 20 seconds. The study included 46 HCWs and 29 patients and lasted for 4 days and 4 nights. 14037 contacts were recorded. The number and duration of contacts varied between mornings, afternoons and nights, and contact matrices describing the mixing patterns between HCW and patients were built for each time period. Contact patterns were qualitatively similar from one day to the next. 38% of the contacts occurred between pairs of HCWs and 6 HCWs accounted for 42% of all the contacts including at least one patient, suggesting a population of individuals who could potentially act as super-spreaders. Wearable sensors represent a novel tool for the measurement of contact patterns in hospitals. The collected data provides information on important aspects that impact the spreading patterns of infectious diseases, such as the strong heterogeneity of contact numbers and durations across individuals, the variability in the number of contacts during a day, and the fraction of repeated contacts across days. This variability is associated with a marked statistical stability of contact and mixing patterns across days. Our results highlight the need for such measurement efforts in order to correctly inform mathematical models of HAIs and use them to inform the design and evaluation of prevention strategies.

Data on face-to-face contacts in an office building suggests a low-cost vaccination strategy based on community linkers

Abstract: Empirical data on contacts between individuals in social contexts play an important role in providing information for models describing human behavior and how epidemics spread in populations. Here, we analyze data on face-to-face contacts collected in an office building. The statistical properties of contacts are similar to other social situations, but important differences are observed in the contact network structure. In particular, the contact network is strongly shaped by the organization of the offices in departments, which has consequences in the design of accurate agent-based models of epidemic spread. We consider the contact network as a potential substrate for infectious disease spread and show that its sparsity tends to prevent outbreaks of rapidly spreading epidemics. Moreover, we define three typical behaviors according to the fraction $f$ of links each individual shares outside its own department: residents, wanderers and linkers. Linkers ($f\sim 50\%$) act as bridges in the network and have large betweenness centralities. Thus, a vaccination strategy targeting linkers efficiently prevents large outbreaks. As such a behavior may be spotted a priori in the offices' organization or from surveys, without the full knowledge of the time-resolved contact network, this result may help the design of efficient, low-cost vaccination or social-distancing strategies.

Simulated Epidemics in an Empirical Spatiotemporal Network of 50,185 Sexual Contacts

Abstract: Sexual contact patterns, both in their temporal and network structure, can influence the spread of sexually transmitted infections (STI). Most previous literature has focused on effects of network topology; few studies have addressed the role of temporal structure. We simulate disease spread using SI and SIR models on an empirical temporal network of sexual contacts in high-end prostitution. We compare these results with several other approaches, including randomization of the data, classic mean-field approaches, and static network simulations. We observe that epidemic dynamics in this contact structure have well-defined, rather high epidemic thresholds. Temporal effects create a broad distribution of outbreak sizes, even if the per-contact transmission probability is taken to its hypothetical maximum of 100%. In general, we conclude that the temporal correlations of our network accelerate outbreaks, especially in the early phase of the epidemics, while the network topology (apart from the contact-rate distribution) slows them down. We find that the temporal correlations of sexual contacts can significantly change simulated outbreaks in a large empirical sexual network. Thus, temporal structures are needed alongside network topology to fully understand the spread of STIs. On a side note, our simulations further suggest that the specific type of commercial sex we investigate is not a reservoir of major importance for HIV.

Running times in seconds for eight distinct networks.

Tab. {{table:e6a4d3c4-0b68-4b59-a46f-fa7896583339}}  depicts our average running time of 10 executions for every procedure, i.e., the average running time needed to open the dataset (1), compute the suggestion curve (2), and compute the colored barcode for a given resolution (3). The experiments were performed on a personal computer with Intel(R) Core(TM) i5-8350U x 8 CPU @ 3.60GHz, 16 GB RAM and Ubuntu 22.10.
The table considers the same eight networks and maximal time values as in Tab. {{table:909af3fa-1aa0-4b15-b494-806ca29f7a1a}}  .


Apply GANs for black-box API attacks to deal with limited training data.

Propose a GANs based intrusion detection system to counter imbalanced learning.

GANs is used to generate flow-based network traffic.

Generative Adversarial Networks for Black-Box API Attacks with Limited Training Data

Abstract: As online systems based on machine learning are offered to public or paid subscribers via application programming interfaces (APIs), they become vulnerable to frequent exploits and attacks. This paper studies adversarial machine learning in the practical case when there are rate limitations on API calls. The adversary launches an exploratory (inference) attack by querying the API of an online machine learning system (in particular, a classifier) with input data samples, collecting returned labels to build up the training data, and training an adversarial classifier that is functionally equivalent and statistically close to the target classifier. The exploratory attack with limited training data is shown to fail to reliably infer the target classifier of a real text classifier API that is available online to the public. In return, a generative adversarial network (GAN) based on deep learning is built to generate synthetic training data from a limited number of real training data samples, thereby extending the training data and improving the performance of the inferred classifier. The exploratory attack provides the basis to launch the causative attack (that aims to poison the training process) and evasion attack (that aims to fool the classifier into making wrong decisions) by selecting training and test data samples, respectively, based on the confidence scores obtained from the inferred classifier. These stealth attacks with small footprint (using a small number of API calls) make adversarial machine learning practical under the realistic case with limited training data available to the adversary.

G-IDS: Generative Adversarial Networks Assisted Intrusion Detection System

Abstract: The boundaries of cyber-physical systems (CPS) and the Internet of Things (IoT) are converging together day by day to introduce a common platform on hybrid systems. Moreover, the combination of artificial intelligence (AI) with CPS creates a new dimension of technological advancement. All these connectivity and dependability are creating massive space for the attackers to launch cyber attacks. To defend against these attacks, intrusion detection system (IDS) has been widely used. However, emerging CPS fields suffer from imbalanced and missing sample data, which makes the training of IDS difficult. In this paper, we propose a generative adversarial network (GAN) based intrusion detection system (G-IDS), where GAN generates synthetic samples, and IDS gets trained on them along with the original ones. G-IDS also fixes the difficulties of imbalanced or missing data problems. We model a network security dataset for an emerging CPS using NSL KDD-99 dataset and evaluate our proposed model's performance using different metrics. We find that our proposed G-IDS model performs much better in attack detection and model stabilization during the training process than a standalone IDS.

Flow-based Network Traffic Generation using Generative Adversarial Networks

Abstract: Flow-based data sets are necessary for evaluating network-based intrusion detection systems (NIDS). In this work, we propose a novel methodology for generating realistic flow-based network traffic. Our approach is based on Generative Adversarial Networks (GANs) which achieve good results for image generation. A major challenge lies in the fact that GANs can only process continuous attributes. However, flow-based data inevitably contain categorical attributes such as IP addresses or port numbers. Therefore, we propose three different preprocessing approaches for flow-based data in order to transform them into continuous values. Further, we present a new method for evaluating the generated flow-based network traffic which uses domain knowledge to define quality tests. We use the three approaches for generating flow-based network traffic based on the CIDDS-001 data set. Experiments indicate that two of the three approaches are able to generate high quality data.

Recent works that use GANs as data oversampler in the security domain (as shown in column three, none of these prior works explored hyperparameter optimization on GANs). Training a GANs model is a difficult task since we have to achieve a balance between its internal components (i.e., the generator and discriminator) {{cite:a9ab837}}. This paper explores this task with hyperparameter optimization with the novel Bayesian optimization.

SMOTE was first proposed in 2002, and this paper explores “can we do better than SMOTE?”. For example, a new approach to generate samples for resampling purposes is GANs {{cite:12275f9}}; i.e. Generative Adversarial Networks. Unlike SMOTE's issue with local inference, GANs oversampling can effectively learn the whole data characteristics and generate samples close to the distribution of original input data. Considering that GANs can achieve some impressive results in producing meaningful, realistic samples in prior studies (e.g., in domains such as computer vision {{cite:f74396e}}, {{cite:c13186c}}), more security practitioners have adopted variants of GANs in many security tasks {{cite:eddd314}}, {{cite:4c2f58a}}, {{cite:72a944b}} (see Table {{table:d091e172-d8b1-4911-ba51-406cc1fc48a9}} ).

GANs are rapidly evolving fields, delivering promising results in generating realistic examples across a range of problem domains, most notably in images tasks such as synthesizing images from text description {{cite:b580320}}, image compression {{cite:9b0f2a2}}, image classification {{cite:2af3f97}}, etc. In the security domain, prior work indicate that GAN would be an ideal technique to train a classification model to explore unforeseen data threats with generated data. Table {{table:d091e172-d8b1-4911-ba51-406cc1fc48a9}}  lists recent work that use GAN as data oversampler. Those works motivate our study, however, we also note that they hardly introduce any way to optimize their GANs architecture as we do in this study.


Emergency management using computer vision

Human centered, vehicle centered and small area centered studies; Methods of detection: Single person, multiple person, vehicles, multi-view camera based.

Emergency classification: Natural, human made (road accident, crowd related, weapon threat, drowning, injured person, falling person); Monitoring objective: Prevention, detection, response and understanding; Acquisition methods: Sensor location, sensor types, acquisition rate and sensor cost; Feature extraction algorithms: Color, shape and texture, temporal (wavelet, optical flow, background modeling and subtraction, tracking) and convolution features; Semantic information extraction using machine learning: Artificial neural networks, deep learning, support vector machines (SVMs), hidden markov models (HMMs), fuzzy logic.

Sudden Event Recognition: A Survey

Abstract: Event recognition is one of the most active research areas in video surveillance fields. Advancement in event recognition systems mainly aims to provide convenience, safety and an efficient lifestyle for humanity. A precise, accurate and robust approach is necessary to enable event recognition systems to respond to sudden changes in various uncontrolled environments, such as the case of an emergency, physical threat and a fire or bomb alert. The performance of sudden event recognition systems depends heavily on the accuracy of low level processing, like detection, recognition, tracking and machine learning algorithms. This survey aims to detect and characterize a sudden event, which is a subset of an abnormal event in several video surveillance applications. This paper discusses the following in detail: (1) the importance of a sudden event over a general anomalous event; (2) frameworks used in sudden event recognition; (3) the requirements and comparative studies of a sudden event recognition system and (4) various decision-making approaches for sudden event recognition. The advantages and drawbacks of using 3D images from multiple cameras for real-time application are also discussed. The paper concludes with suggestions for future research directions in sudden event recognition.

Review on Computer Vision Techniques in Emergency Situation

Abstract: In emergency situations, actions that save lives and limit the impact of hazards are crucial. In order to act, situational awareness is needed to decide what to do. Geolocalized photos and video of the situations as they evolve can be crucial in better understanding them and making decisions faster. Cameras are almost everywhere these days, either in terms of smartphones, installed CCTV cameras, UAVs or others. However, this poses challenges in big data and information overflow. Moreover, most of the time there are no disasters at any given location, so humans aiming to detect sudden situations may not be as alert as needed at any point in time. Consequently, computer vision tools can be an excellent decision support. The number of emergencies where computer vision tools has been considered or used is very wide, and there is a great overlap across related emergency research. Researchers tend to focus on state-of-the-art systems that cover the same emergency as they are studying, obviating important research in other fields. In order to unveil this overlap, the survey is divided along four main axes: the types of emergencies that have been studied in computer vision, the objective that the algorithms can address, the type of hardware needed and the algorithms used. Therefore, this review provides a broad overview of the progress of computer vision covering all sorts of emergencies.

Surveys on computer vision-based methods in surveillance

During last 10 years or so, a few interesting surveys have been published in this field of research.
Authors of {{cite:0370237}} have explored object detection, tracking, scene modeling and activity analysis using video trajectories. The study presented in {{cite:7222f78}} covers vehicle detection, tracking, behavior understanding and incident detection from the purview of intelligent transportation systems (ITS). Authors of {{cite:2cdf5c6}} have conducted an in-depth study of traffic analysis frameworks under different taxonomies with pointers at integrating information from multiple sensors. The review presented in {{cite:d35606e}} is possibly the first work covering anomaly detection techniques. It covers sensors, entities, feature extraction methods, learning methods and scene modeling to detect anomalies. In {{cite:4cf05b8}}, an object oriented approach from the perspective of vehicle mounted sensors for object detection, tracking and behavior analysis detailing the progress of the last decade of works, has been presented. Multi-camera study presented in {{cite:a308598}} covers the researches related to surveillance in multi-camera setups. Authors of {{cite:2fb264a}} discuss events, which are considered as a subset of anomalous events, requiring immediate attention, occuring unintentionally, abruptly and unexpectedly. The research presented in {{cite:a96044a}} discusses safety, security and law enforcement related applications from the computer vision perspective. The review presented in {{cite:dbb400b}} discusses the elements of human activity and behavioral understanding frameworks. Authors of {{cite:6125f7b}} present the researches on human behavioral understanding through actions and interactions of human entities. Intelligent video systems covering analytics aspect has been studied in {{cite:36974ee}}. Surveillance systems with specific application areas have been presented in {{cite:f8bbeab}}. Authors of {{cite:23255e6}} systematically divide road traffic analysis into four layers, namely image acquisition, dynamic and static attribute extraction, behavioral understanding and ITS services. Datasets used for anomaly detections have been covered in {{cite:5ea4990}}. Traffic monitoring using different types of sensors has been discussed in {{cite:2d9b6cd}}. Algorithms used for spatio-temporal point detections and their applications in vision domain have been covered in {{cite:bc7bcc7}}. Traffic entities have been studied from the perspective of safety in {{cite:c508b8c}}. Authors of {{cite:a6f872b}} explore studies on video trajectory-based analysis and applications. Authors of {{cite:c2ab244}} discuss various ways of handling emergency situations by assessing the risks, preparedness, response, recovery and mitigation using the extracted information from the visual features with the help of various learning mechanisms. In {{cite:26fd5f8}}, authors have presented anomalous human behavior recognition work with focus on behavior representation and modeling, feature extraction techniques, classification and behavior modeling frameworks, performance evaluation techniques, and datasets with examples of video surveillance systems. Table {{table:019a5bf1-06bc-41a1-806e-6803f676b83b}}  summarizes the major computer vision-based studies done during last 10 years. In our survey, we particularly focus on the studies on anomaly detection that are relevant on road traffic scenarios.


Open Quantum Assembly Language

Abstract: This document describes a quantum assembly language (QASM) called OpenQASM that is used to implement experiments with low depth quantum circuits. OpenQASM represents universal physical circuits over the CNOT plus SU(2) basis with straight-line code that includes measurement, reset, fast feedback, and gate subroutines. The simple text language can be written by hand or by higher level tools and may be executed on the IBM Q Experience.

Open Quantum Assembly Language

Open Quantum Assembly Language

Open Quantum Assembly Language

Open Quantum Assembly Language

Open Quantum Assembly Language

QASMBench: A Low-level QASM Benchmark Suite for NISQ Evaluation and Simulation

Abstract: The rapid development of quantum computing (QC) in the NISQ era urgently demands a low-level benchmark suite and insightful evaluation metrics for characterizing the properties of prototype NISQ devices, the efficiency of QC programming compilers, schedulers and assemblers, and the capability of quantum system simulators in a classical computer. In this work, we fill this gap by proposing a low-level, easy-to-use benchmark suite called QASMBench based on the OpenQASM assembly representation. It consolidates commonly used quantum routines and kernels from a variety of domains including chemistry, simulation, linear algebra, searching, optimization, arithmetic, machine learning, fault tolerance, cryptography, etc., trading-off between generality and usability. To analyze these kernels in terms of NISQ device execution, in addition to circuit width and depth, we propose four circuit metrics including gate density, retention lifespan, measurement density, and entanglement variance, to extract more insights about the execution efficiency, the susceptibility to NISQ error, and the potential gain from machine-specific optimizations. Applications in QASMBench can be launched and verified on several NISQ platforms, including IBM-Q, Rigetti, IonQ and Quantinuum. For evaluation, we measure the execution fidelity of a subset of QASMBench applications on 12 IBM-Q machines through density matrix state tomography, which comprises 25K circuit evaluations. We also compare the fidelity of executions among the IBM-Q machines, the IonQ QPU and the Rigetti Aspen M-1 system. QASMBench is released at: http://github.com/pnnl/QASMBench.

blackQASMBench Large-scale Benchmarks.

For each item in Table {{table:ef526f5f-b131-4e09-bc0e-b1e0f6d3b5d3}} , {{table:7a1bdd5c-0b3e-4da8-b4a4-9c5076eb5991}}  and {{table:5e9bb705-ae98-4c14-8b25-47d9c996fd0c}} , we list its name, brief description, and the algorithm category it belongs to in the quantum stack (see Figure {{figure:b74eb930-cba1-4979-9775-2d8949b709c1}} ). We show the number of qubits and gates utilized in the routines. The gate number here refers to standard OpenQASM gates (not basic gates or composition gates) as discussed in Section 1.2. We also list the number of CX gates in the tables, which indicates a major source of delay and error. Each algorithm is briefly introduced under its respective category. However, certain routines appear in multiple categories, we only describe them in the first appearance. Last not the least, although QASMbench is developed based on OpenQASM, they can be converted to other representations such as Q#, PyQuil, Cirq, etc. through the Javascript based q-convert tool, which is available online: http://quantum-circuit.com/qconvert. blackWe distribute QASMBench in other representations as well through a modified version of q-convert.