Exploratory, Evasion, Causative, Black box

Launch an over-the-air attack to infer the transmitter's behavior and falsify the spectrum sensing data by transmitting perturbations over the air.

Build a DNN classifier to infer the channel access algorithm of a transmitter and predict the transmission result, and develop a defense mechanism to confuse the adversary in its attack strategy.

Propose Learning-Evaluation-Beating(LEB) attack against cooperative spectrum sensing (extension of spectrum sensing data falsification attack on cognitive radio).

Launch over-the-air spectrum poisoning attacks by learning first the transmitter's behavior.

Present AML attack on spectrum sharing of 5G communications with incumbent users.

Jamming, Spectrum poisoning, Priority violation

Attacker jams the transmitter's sensing period

Manipulate inputs to the classifier by transmitting during spectrum sensing

Dataset collected from 5282 locations {{cite:3a0e7f3}}, Data from 5 locations using USRP N210

Spectrum Data Poisoning with Adversarial Deep Learning

Abstract: Machine learning has been widely applied in wireless communications. However, the security aspects of machine learning in wireless applications have not been well understood yet. We consider the case that a cognitive transmitter senses the spectrum and transmits on idle channels determined by a machine learning algorithm. We present an adversarial machine learning approach to launch a spectrum data poisoning attack by inferring the transmitter's behavior and attempting to falsify the spectrum sensing data over the air. For that purpose, the adversary transmits for a short period of time when the channel is idle to manipulate the input for the decision mechanism of the transmitter. The cognitive engine at the transmitter is a deep neural network model that predicts idle channels with minimum sensing error for data transmissions. The transmitter collects spectrum sensing data and uses it as the input to its machine learning algorithm. In the meantime, the adversary builds a cognitive engine using another deep neural network model to predict when the transmitter will have a successful transmission based on its spectrum sensing data. The adversary then performs the over-the-air spectrum data poisoning attack, which aims to change the channel occupancy status from idle to busy when the transmitter is sensing, so that the transmitter is fooled into making incorrect transmit decisions. This attack is more energy efficient and harder to detect compared to jamming of data transmissions. We show that this attack is very effective and reduces the throughput of the transmitter substantially.

IoT Network Security from the Perspective of Adversarial Deep Learning

Abstract: Machine learning finds rich applications in Internet of Things (IoT) networks such as information retrieval, traffic management, spectrum sensing, and signal authentication. While there is a surge of interest to understand the security issues of machine learning, their implications have not been understood yet for wireless applications such as those in IoT systems that are susceptible to various attacks due the open and broadcast nature of wireless communications. To support IoT systems with heterogeneous devices of different priorities, we present new techniques built upon adversarial machine learning and apply them to three types of over-the-air (OTA) wireless attacks, namely denial of service (DoS) attack in terms of jamming, spectrum poisoning attack, and priority violation attack. By observing the spectrum, the adversary starts with an exploratory attack to infer the channel access algorithm of an IoT transmitter by building a deep neural network classifier that predicts the transmission outcomes. Based on these prediction results, the wireless attack continues to either jam data transmissions or manipulate sensing results over the air (by transmitting during the sensing phase) to fool the transmitter into making wrong transmit decisions in the test phase (corresponding to an evasion attack). When the IoT transmitter collects sensing results as training data to retrain its channel access algorithm, the adversary launches a causative attack to manipulate the input data to the transmitter over the air. We show that these attacks with different levels of energy consumption and stealthiness lead to significant loss in throughput and success ratio in wireless communications for IoT systems. Then we introduce a defense mechanism that systematically increases the uncertainty of the adversary at the inference stage and improves the performance. Results provide new insights on how to attack and defend IoT networks using deep learning.

When Attackers Meet AI: Learning-Empowered Attacks in Cooperative Spectrum Sensing

Abstract: Defense strategies have been well studied to combat Byzantine attacks that aim to disrupt cooperative spectrum sensing by sending falsified versions of spectrum sensing data to a fusion center. However, existing studies usually assume network or attackers as passive entities, e.g., assuming the prior knowledge of attacks is known or fixed. In practice, attackers can actively adopt arbitrary behaviors and avoid pre-assumed patterns or assumptions used by defense strategies. In this paper, we revisit this security vulnerability as an adversarial machine learning problem and propose a novel learning-empowered attack framework named Learning-Evaluation-Beating (LEB) to mislead the fusion center. Based on the black-box nature of the fusion center in cooperative spectrum sensing, our new perspective is to make the adversarial use of machine learning to construct a surrogate model of the fusion center's decision model. We propose a generic algorithm to create malicious sensing data using this surrogate model. Our real-world experiments show that the LEB attack is effective to beat a wide range of existing defense strategies with an up to 82 percent of success ratio. Given the gap between the proposed LEB attack and existing defenses, we introduce a non-invasive method named as influence-limiting defense, which can coexist with existing defenses to defend against LEB attack or other similar attacks. We show that this defense is highly effective and reduces the overall disruption ratio of LEB attack by up to 80 percent.

Adversarial Deep Learning for Over-the-Air Spectrum Poisoning Attacks

Abstract: An adversarial deep learning approach is presented to launch over-the-air spectrum poisoning attacks. A transmitter applies deep learning on its spectrum sensing results to predict idle time slots for data transmission. In the meantime, an adversary learns the transmitter's behavior (exploratory attack) by building another deep neural network to predict when transmissions will succeed. The adversary falsifies (poisons) the transmitter's spectrum sensing data over the air by transmitting during the short spectrum sensing period of the transmitter. Depending on whether the transmitter uses the sensing results as test data to make transmit decisions or as training data to retrain its deep neural network, either it is fooled into making incorrect decisions (evasion attack) or the transmitter's algorithm is retrained incorrectly for future decisions (causative attack). Both attacks are energy efficient and hard to detect (stealth) compared to jamming the long data transmission period, and substantially reduce the throughput. A dynamic defense is designed for the transmitter that deliberately makes a small number of incorrect transmissions (selected by the confidence score on channel classification) to manipulate the adversary's training data. This defense effectively fools the adversary (if any) and helps the transmitter sustain its throughput with or without an adversary present.

Adversarial Machine Learning for 5G Communications Security

Abstract: Machine learning provides automated means to capture complex dynamics of wireless spectrum and support better understanding of spectrum resources and their efficient utilization. As communication systems become smarter with cognitive radio capabilities empowered by machine learning to perform critical tasks such as spectrum awareness and spectrum sharing, they also become susceptible to new vulnerabilities due to the attacks that target the machine learning applications. This paper identifies the emerging attack surface of adversarial machine learning and corresponding attacks launched against wireless communications in the context of 5G systems. The focus is on attacks against (i) spectrum sharing of 5G communications with incumbent users such as in the Citizens Broadband Radio Service (CBRS) band and (ii) physical layer authentication of 5G User Equipment (UE) to support network slicing. For the first attack, the adversary transmits during data transmission or spectrum sensing periods to manipulate the signal-level inputs to the deep learning classifier that is deployed at the Environmental Sensing Capability (ESC) to support the 5G system. For the second attack, the adversary spoofs wireless signals with the generative adversarial network (GAN) to infiltrate the physical layer authentication mechanism based on a deep learning classifier that is deployed at the 5G base station. Results indicate major vulnerabilities of 5G systems to adversarial machine learning. To sustain the 5G system operations in the presence of adversaries, a defense mechanism is presented to increase the uncertainty of the adversary in training the surrogate model used for launching its subsequent attacks.

Review of previous works on AML attacks in spectrum sensing.

Review of Adversarial Machine Learning in Wireless Communications

Grouped according to the type of DL problem, we discuss and review various studies on AML in wireless communication and present them in Tables {{table:6d0818f9-5311-467b-82ca-317c3ca1bda6}}  (for modulation classification), Table {{table:74faa1c0-f85e-4043-83bf-1ce6fb7f8b09}}  (for spectrum sensing), Table {{table:84381566-e709-4759-83e0-8cfc61cc164c}}  (for signal classification), and Table {{table:e02b4b9b-d511-40b8-ba3e-99d621d4276a}}  (for other areas in the wireless communication domain).

Most of the AML studies focused on spectrum sensing and dynamic spectrum access (DSA) use exploratory attacks in which the attacker learns the operational pattern of the DNN model and builds a surrogate model which gives the same or similar output as the original classifier when given the same input. Spectrum sensing and DSA are key components of cognitive radio systems for efficient discovery and use of spectrum to achieve situational awareness {{cite:3a438ef}}. Given the broadcast and over-the-air nature of wireless communications, there are various attacks that attempt to undermine the spectrum sensing results. This can include causative attack to change the sensed features as well as priority violation attacks to violate priorities in channel access by pretending to have higher priority and transmitting during the sensing phase {{cite:32e4e33}}. AML attacks on spectrum sensing typically involve an adversary that develops a surrogate model and can correctly predict the outcome of the original DL model. Such an adversary jams the successful transmissions, thus reducing the success rate of the original model {{cite:9133e58}}, {{cite:437e679}}, {{cite:ac31527}}. As shown in {{cite:9133e58}}, {{cite:437e679}}, {{cite:78bc517}}, {{cite:7af5049}}, these attacks are more energy-efficient and more effective than random jamming. Table {{table:74faa1c0-f85e-4043-83bf-1ce6fb7f8b09}}  highlight the studies on AML for spectrum sensing and DSA applications.


ViZDoom: A Doom-based AI research platform for visual reinforcement learning

Abstract: The recent advances in deep neural networks have led to effective vision-based reinforcement learning methods that have been employed to obtain human-level controllers in Atari 2600 games from pixel data. Atari 2600 games, however, do not resemble real-world tasks since they involve non-realistic 2D environments and the third-person perspective. Here, we propose a novel test-bed platform for reinforcement learning research from raw visual information which employs the first-person perspective in a semi-realistic 3D world. The software, called ViZDoom, is based on the classical first-person shooter video game, Doom. It allows developing bots that play the game using the screen buffer. ViZDoom is lightweight, fast, and highly customizable via a convenient mechanism of user scenarios. In the experimental part, we test the environment by trying to learn bots for two scenarios: a basic move-and-shoot task and a more complex maze-navigation problem. Using convolutional deep neural networks with Q-learning and experience replay, for both scenarios, we were able to train competent bots, which exhibit human-like behaviors. The results confirm the utility of ViZDoom as an AI research platform and imply that visual reinforcement learning in 3D realistic first-person perspective environments is feasible.

DeepMind Lab

Abstract: DeepMind Lab is a first-person 3D game platform designed for research and development of general artificial intelligence and machine learning systems. DeepMind Lab can be used to study how autonomous artificial agents may learn complex tasks in large, partially observed, and visually diverse worlds. DeepMind Lab has a simple and flexible API enabling creative task-designs and novel AI-designs to be explored and quickly iterated upon. It is powered by a fast and widely recognised game engine, and tailored for effective use by the research community.

Abstract: We introduce HoME: a Household Multimodal Environment for artificial agents to learn from vision, audio, semantics, physics, and interaction with objects and other agents, all within a realistic context. HoME integrates over 45,000 diverse 3D house layouts based on the SUNCG dataset, a scale which may facilitate learning, generalization, and transfer. HoME is an open-source, OpenAI Gym-compatible platform extensible to tasks in reinforcement learning, language grounding, sound-based navigation, robotics, multi-agent learning, and more. We hope HoME better enables artificial agents to learn as humans do: in an interactive, multimodal, and richly contextualized setting.

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

Abstract: A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visually grounded sequence-to-sequence translation problems, and many of the same methods are applicable. To enable and encourage the application of vision and language methods to the problem of interpreting visually-grounded navigation instructions, we present the Matter-port3D Simulator - a large-scale reinforcement learning environment based on real imagery [11]. Using this simulator, which can in future support a range of embodied vision and language tasks, we provide the first benchmark dataset for visually-grounded natural language navigation in real buildings - the Room-to-Room (R2R) dataset1.

AI2-THOR: An Interactive 3D Environment for Visual AI

Abstract: We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor.allenai.org. AI2-THOR consists of near photo-realistic 3D indoor scenes, where AI agents can navigate in the scenes and interact with objects to perform tasks. AI2-THOR enables research in many different domains including but not limited to deep reinforcement learning, imitation learning, learning by interaction, planning, visual question answering, unsupervised representation learning, object detection and segmentation, and learning models of cognition. The goal of AI2-THOR is to facilitate building visually intelligent models and push the research forward in this domain.

Comparison of the first person 3D environment simulators available in the RL community. Frames per second (FPS) is shown for a single CPU core.

Environments —
Beyond the environments available in the Gym framework {{cite:fc4ae77}} for Atari and continuous control tasks, there are numerous 3D simulators available with a range advantages and disadvantages. Before starting this work, we investigated the limitations of each simulator, which are summarized in table {{table:cce90f19-0c67-4172-a7aa-26692c97832c}} , as we focus on proxies for in home mobile robots we exclude comparisons with car and drone simulators such as CARLA {{cite:5a6e4dc}} and Sim4CV {{cite:66cdd3d}}. In choosing the simulator, speed was of utmost importance as we did not want to starve the GPU of data. The fidelity and realism of the simulator was not of great concern, as the intention was to run the simulator at a low resolution where fine detail cannot be captured. We chose ViZDoom as it is orders of magnitude faster than its competitors. The downside of ViZDoom is that there are only 8 scenarios available which are limited in scope and cannot be used to evaluate the generalization performance of an agent. For this reason we have constructed an ever growing collection of scenarios that aim to test navigation, reasoning and memorization. Four of the scenarios have been included in the proposed benchmark.


tumor growth, free cell boundaries coupled to a Navier–Stokes solver

viscoelastic polygonal cells with cytoskeletal elements

polygonal cell boundaries immersed in a LBM fluid with chemical signaling

packing of elastic particles with free polygonal boundaries

demonstrated tumor tissue growth to about 900 cells

demonstrated simulations with a few hundred cells

can grow a tissue to 10000 cells in a day

demonstrated simulations with up to 1000 particles

demonstrated simulations with up to 400 cells

A single-cell approach in modeling the dynamics of tumor microregions.

Abstract: Interactions between tumor cells and their environment lead to the formation of microregions containing nonhomogeneous subpopulations of cells and steep gradients in oxygen, glucose, and other metabolites. To address the formation of tumor microregions on the level of single cells, I propose a new two-dimensional time-dependent mathematical model taking explicitly into account the individually regulated biomechanical processes of tumor cells and the effect of oxygen consumption on their metabolism. Numerical simulations of the self-organized formation of tumor microregions are presented and the dynamics of such a process is discussed.

A Sub-Cellular Viscoelastic Model for Cell Population Mechanics

Abstract: Understanding the biomechanical properties and the effect of biomechanical force on epithelial cells is key to understanding how epithelial cells form uniquely shaped structures in two or three-dimensional space. Nevertheless, with the limitations and challenges posed by biological experiments at this scale, it becomes advantageous to use mathematical and ‘in silico’ (computational) models as an alternate solution. This paper introduces a single-cell-based model representing the cross section of a typical tissue. Each cell in this model is an individual unit containing several sub-cellular elements, such as the elastic plasma membrane, enclosed viscoelastic elements that play the role of cytoskeleton, and the viscoelastic elements of the cell nucleus. The cell membrane is divided into segments where each segment (or point) incorporates the cell's interaction and communication with other cells and its environment. The model is capable of simulating how cells cooperate and contribute to the overall structure and function of a particular tissue; it mimics many aspects of cellular behavior such as cell growth, division, apoptosis and polarization. The model allows for investigation of the biomechanical properties of cells, cell-cell interactions, effect of environment on cellular clusters, and how individual cells work together and contribute to the structure and function of a particular tissue. To evaluate the current approach in modeling different topologies of growing tissues in distinct biochemical conditions of the surrounding media, we model several key cellular phenomena, namely monolayer cell culture, effects of adhesion intensity, growth of epithelial cell through interaction with extra-cellular matrix (ECM), effects of a gap in the ECM, tensegrity and tissue morphogenesis and formation of hollow epithelial acini. The proposed computational model enables one to isolate the effects of biomechanical properties of individual cells and the communication between cells and their microenvironment while simultaneously allowing for the formation of clusters or sheets of cells that act together as one complex tissue.

LBIBCell: a cell-based simulation environment for morphogenetic problems

Abstract:
MOTIVATION
The simulation of morphogenetic problems requires the simultaneous and coupled simulation of signalling and tissue dynamics. A cellular resolution of the tissue domain is important to adequately describe the impact of cell-based events, such as cell division, cell-cell interactions and spatially restricted signalling events. A tightly coupled cell-based mechano-regulatory simulation tool is therefore required.

RESULTS
We developed an open-source software framework for morphogenetic problems. The environment offers core functionalities for the tissue and signalling models. In addition, the software offers great flexibility to add custom extensions and biologically motivated processes. Cells are represented as highly resolved, massless elastic polygons; the viscous properties of the tissue are modelled by a Newtonian fluid. The Immersed Boundary method is used to model the interaction between the viscous and elastic properties of the cells, thus extending on the IBCell model. The fluid and signalling processes are solved using the Lattice Boltzmann method. As application examples we simulate signalling-dependent tissue dynamics.

AVAILABILITY AND IMPLEMENTATION
The documentation and source code are available on http://tanakas.bitbucket.org/lbibcell/index.html

Jamming of Deformable Polygons.

Abstract: We introduce the deformable particle (DP) model for cells, foams, emulsions, and other soft particulate materials, which adds to the benefits and eliminates deficiencies of existing models. The DP model combines the ability to model individual soft particles with the shape-energy function of the vertex model, and adds arbitrary particle deformations. We focus on 2D deformable polygons with a shape-energy function that is minimized for area a_{0} and perimeter p_{0} and repulsive interparticle forces. We study the onset of jamming versus particle asphericity, A=p_{0}^{2}/4πa_{0}, and find that the packing fraction grows with A until reaching A^{*}=1.16 of the underlying Voronoi cells at confluence. We find that DP packings above and below A^{*} are solidlike, which helps explain the solid-to-fluid transition at A^{*} in the vertex model as a transition from tension- to compression-dominated regimes.

PalaCell2D: A framework for detailed tissue morphogenesis

Abstract: In silico, cell based approaches for modeling biological morphogenesis are used to test and validate our understanding of the biological and mechanical process that are at work during the growth and the organization of multi-cell tissues. As compared to in vivo experiments, computer based frameworks dedicated to tissue modeling allow us to easily test different hypotheses, and to quantify the impact of various biophysically relevant parameters. Here, we propose a formalism based on a detailed, yet simple, description of cells that accounts for intra-, inter- and extra-cellular mechanisms. More precisely, the cell growth and division is described through the space and time evolution of the membrane vertices. These vertices follow a Newtonian dynamics, meaning that their evolution is con-trolled by different types of forces: a membrane force (spring and bending), an adherence force (inter-cellular spring), external and internal pressure forces. Different evolution laws can be applied on the internal pressure, depending on the intra-cellular mechanism of interest. In addition to the cells dynamics, our formalism further relies on a lattice Boltzmann method, using the Palabos library, to simulate the diffusion of chemical signals. The latter aims at driving the growth and migration of a tissue by simply changing the state of the cells. All of this leads to an accurate description of the growth and division of cells, with realistic cell shapes and where membranes can have different properties. While this work is mainly of methodological nature, we also propose to validate our framework through simple, yet biologically relevant benchmark tests at both single-cell and full tissue scales. This includes free and chemically controlled cell tissue growth in an unbounded domain. The ability of our framework to simulate cell migration, cell compression and morphogenesis under external constraints is also investigated in a qualitative manner.

Overview of related computer programs. Only models in two spatial dimensions and with geometrically represented, curved cell/bubble boundaries are included. Line numbers, runtimes and cell counts are approximate (rounded). Numbers in parentheses indicate the number of lines excluding blanks and comments. OS: Open source. CC: Creative Commons. CPC: Computer Physics Communications. GPL: GNU General Public License. MIT: Massachusetts Institute of Technology. LBM: Lattice Boltzmann Method.

Computer simulations of such systems started mainly in the 1980s with vertex models {{cite:98fea4f}}, {{cite:c60d39a}}, {{cite:dc39f49}}, followed by a program named 2D-FROTH representing dry froths and foams with curved bubble boundaries {{cite:535e2b7}}—a feature also particularly important in tissue biology {{cite:2e937ea}}, {{cite:c3e382a}}, {{cite:3498cc9}}. These early implementations used shared polygonal boundaries between neighboring cells. More topological freedom in the form of individual, free boundaries was introduced with a C program named PLAT shortly after {{cite:459c59e}}, {{cite:60fa2a4}}. In the 2000s and 2010s, several new two-dimensional mechanical models were developed to simulate a variety of phenomena including tissue growth and morphogenesis, cell migration and aggregation, soft particle packing, and many more. We review the history of developed computer programs that continued along the path of polygonal representations of fluidic domains in Table {{table:b5aa7636-e375-4c13-b8ae-97e1ed6c31ef}} . About half of these computational models were made open-source, published under various licenses, with a tendency in the last decade toward more openness, but also more copyleft.

Also belonging to the class or amorphous solids, but worth mentioning separately here for their historical significance (cf. Table {{table:b5aa7636-e375-4c13-b8ae-97e1ed6c31ef}} ), are foams and emulsions. A relatively wet foam is shown in Fig. {{figure:de2af48c-167f-48df-a43c-9b8ac55a6869}} F, produced by letting a polydisperse collection of nearly incompressible bubbles, initially circular in shape and randomly placed in the plane, drop under the action of gravity. Governed by surface tension and weak cohesion (Table {{table:4ba5dbc2-4ad8-436b-ab06-e59f69982d5d}} ), the bubbles deform and coalesce into a foamy structure with a free upper surface (Movie 4).


"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection

Abstract: Automatic fake news detection is a challenging problem in deception detection, and it has tremendous real-world political and social impacts. However, statistical approaches to combating fake news has been dramatically limited by the lack of labeled benchmark datasets. In this paper, we present liar: a new, publicly available dataset for fake news detection. We collected a decade-long, 12.8K manually labeled short statements in various contexts from PolitiFact.com, which provides detailed analysis report and links to source documents for each case. This dataset can be used for fact-checking research as well. Notably, this new dataset is an order of magnitude larger than previously largest public fake news datasets of similar type. Empirically, we investigate automatic fake news detection based on surface-level linguistic patterns. We have designed a novel, hybrid convolutional neural network to integrate meta-data with text. We show that this hybrid approach can improve a text-only deep learning model.

Integrating Stance Detection and Fact Checking in a Unified Corpus

Abstract: A reasonable approach for fact checking a claim involves retrieving potentially relevant documents from different sources (e.g., news websites, social media, etc.), determining the stance of each document with respect to the claim, and finally making a prediction about the claim's factuality by aggregating the strength of the stances, while taking the reliability of the source into account. Moreover, a fact checking system should be able to explain its decision by providing relevant extracts (rationales) from the documents. Yet, this setup is not directly supported by existing datasets, which treat fact checking, document retrieval, source credibility, stance detection and rationale extraction as independent tasks. In this paper, we support the interdependencies between these tasks as annotations in the same corpus. We implement this setup on an Arabic fact checking corpus, the first of its kind.

A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking

Abstract: Automated fact-checking based on machine learning is a promising approach to identify false information distributed on the web. In order to achieve satisfactory performance, machine learning methods require a large corpus with reliable annotations for the different tasks in the fact-checking process. Having analyzed existing fact-checking corpora, we found that none of them meets these criteria in full. They are either too small in size, do not provide detailed annotations, or are limited to a single domain. Motivated by this gap, we present a new substantially sized mixed-domain corpus with annotations of good quality for the core fact-checking tasks: document retrieval, evidence extraction, stance detection, and claim validation. To aid future corpus construction, we describe our methodology for corpus creation and annotation, and demonstrate that it results in substantial inter-annotator agreement. As baselines for future research, we perform experiments on our corpus with a number of model architectures that reach high performance in similar problem settings. Finally, to support the development of future models, we provide a detailed error analysis for each of the tasks. Our results show that the realistic, multi-domain setting defined by our data poses new challenges for the existing models, providing opportunities for considerable improvement by future systems.

MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims

Abstract: We contribute the largest publicly available dataset of naturally occurring factual claims for the purpose of automatic claim verification. It is collected from 26 fact checking websites in English, paired with textual sources and rich metadata, and labelled for veracity by human expert journalists. We present an in-depth analysis of the dataset, highlighting characteristics and challenges. Further, we present results for automatic veracity prediction, both with established baselines and with a novel method for joint ranking of evidence pages and predicting veracity that outperforms all baselines. Significant performance increases are achieved by encoding evidence, and by modelling metadata. Our best-performing model achieves a Macro F1 of 49.2%, showing that this is a challenging testbed for claim veracity prediction.

CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims

Abstract: We introduce CLIMATE-FEVER, a new publicly available dataset for verification of climate change-related claims. By providing a dataset for the research community, we aim to facilitate and encourage work on improving algorithms for retrieving evidential support for climate-specific claims, addressing the underlying language understanding challenges, and ultimately help alleviate the impact of misinformation on climate change. We adapt the methodology of FEVER [1], the largest dataset of artificially designed claims, to real-life claims collected from the Internet. While during this process, we could rely on the expertise of renowned climate scientists, it turned out to be no easy task. We discuss the surprising, subtle complexity of modeling real-world climate-related claims within the \textsc{fever} framework, which we believe provides a valuable challenge for general natural language understanding. We hope that our work will mark the beginning of a new exciting long-term joint effort by the climate science and AI community.

Fact or Fiction: Verifying Scientific Claims

Abstract: We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim, and to identify rationales justifying each decision. To study this task, we construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. We develop baseline models for SciFact, and demonstrate that simple domain adaptation techniques substantially improve performance compared to models trained on Wikipedia or political news. We show that our system is able to verify claims related to COVID-19 by identifying evidence from the CORD-19 corpus. Our experiments indicate that SciFact will provide a challenging testbed for the development of new systems designed to retrieve and reason over corpora containing specialized domain knowledge. Data and code for this new task are publicly available at https://github.com/allenai/scifact. A leaderboard and COVID-19 fact-checking demo are available at https://scifact.apps.allenai.org.

Explainable Automated Fact-Checking for Public Health Claims

Abstract: Fact-checking is the task of verifying the veracity of claims by assessing their assertions against credible evidence. The vast majority of fact-checking studies focus exclusively on political claims. Very little research explores fact-checking for other topics, specifically subject matters for which expertise is required. We present the first study of explainable fact-checking for claims which require specific expertise. For our case study we choose the setting of public health. To support this case study we construct a new dataset PUBHEALTH of 11.8K claims accompanied by journalist crafted, gold standard explanations (i.e., judgments) to support the fact-check labels for claims. We explore two tasks: veracity prediction and explanation generation. We also define and evaluate, with humans and computationally, three coherence properties of explanation quality. Our results indicate that, by training on in-domain data, gains can be made in explainable, automated fact-checking for claims which require specific expertise.

COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic

Abstract: We introduce a FEVER-like dataset COVID-Fact of $4,086$ claims concerning the COVID-19 pandemic. The dataset contains claims, evidence for the claims, and contradictory claims refuted by the evidence. Unlike previous approaches, we automatically detect true claims and their source articles and then generate counter-claims using automatic methods rather than employing human annotators. Along with our constructed resource, we formally present the task of identifying relevant evidence for the claims and verifying whether the evidence refutes or supports a given claim. In addition to scientific claims, our data contains simplified general claims from media sources, making it better suited for detecting general misinformation regarding COVID-19. Our experiments indicate that COVID-Fact will provide a challenging testbed for the development of new systems and our approach will reduce the costs of building domain-specific datasets for detecting misinformation.

X-FACT: A New Benchmark Dataset for Multilingual Fact Checking

Abstract: In this work, we introduce X-FACT: the largest publicly available multilingual dataset for factual verification of naturally existing real-world claims. The dataset contains short statements in 25 languages and is labeled for veracity by expert fact-checkers. The dataset includes a multilingual evaluation benchmark that measures both out-of-domain generalization, and zero-shot capabilities of the multilingual models. Using state-of-the-art multilingual transformer-based models, we develop several automated fact-checking models that, along with textual claims, make use of additional metadata and evidence from news stories retrieved using a search engine. Empirically, our best model attains an F-score of around 40%, suggesting that our dataset is a challenging benchmark for evaluation of multilingual fact-checking models.

Fact Checking in Community Forums

Abstract: Community Question Answering (cQA) forums are very popular nowadays, as they represent effective means for communities around particular topics to share information. Unfortunately, this information is not always factual. Thus, here we explore a new dimension in the context of cQA, which has been ignored so far: checking the veracity of answers to particular questions in cQA forums. As this is a new problem, we create a specialized dataset for it. We further propose a novel multi-faceted model, which captures information from the answer content (what is said and how), from the author profile (who says it), from the rest of the community forum (where it is said), and from external authoritative sources of information (external support). Evaluation results show a MAP value of 86.54, which is 21 points absolute above the baseline.

A Stylometric Inquiry into Hyperpartisan and Fake News

Abstract: We report on a comparative style analysis of hyperpartisan (extremely one-sided) news and fake news. A corpus of 1,627 articles from 9 political publishers, three each from the mainstream, the hyperpartisan left, and the hyperpartisan right, have been fact-checked by professional journalists at BuzzFeed: 97% of the 299 fake news articles identified are also hyperpartisan. We show how a style analysis can distinguish hyperpartisan news from the mainstream (F1 = 0.78), and satire from both (F1 = 0.81). But stylometry is no silver bullet as style-based fake news detection does not work (F1 = 0.46). We further reveal that left-wing and right-wing news share significantly more stylistic similarities than either does with the mainstream. This result is robust: it has been confirmed by three different modeling approaches, one of which employs Unmasking in a novel way. Applications of our results include partisanship detection and pre-screening for semi-automatic fake news detection.

FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media

Abstract: Social media has become a popular means for people to consume news. Meanwhile, it also enables the wide dissemination of fake news, i.e., news with intentionally false information, which brings significant negative effects to the society. Thus, fake news detection is attracting increasing attention. However, fake news detection is a non-trivial task, which requires multi-source information such as news content, social context, and dynamic information. First, fake news is written to fool people, which makes it difficult to detect fake news simply based on news contents. In addition to news contents, we need to explore social contexts such as user engagements and social behaviors. For example, a credible user's comment that "this is a fake news" is a strong signal for detecting fake news. Second, dynamic information such as how fake news and true news propagate and how users' opinions toward news pieces are very important for extracting useful patterns for (early) fake news detection and intervention. Thus, comprehensive datasets which contain news content, social context, and dynamic information could facilitate fake news propagation, detection, and mitigation; while to the best of our knowledge, existing datasets only contains one or two aspects. Therefore, in this paper, to facilitate fake news related researches, we provide a fake news data repository FakeNewsNet, which contains two comprehensive datasets that includes news content, social context, and dynamic information. We present a comprehensive description of datasets collection, demonstrate an exploratory analysis of this data repository from different perspectives, and discuss the benefits of FakeNewsNet for potential applications on fake news study on social media.

FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19

Abstract: In this paper, we present a first multilingual cross-domain dataset of 5182 fact-checked news articles for COVID-19, collected from 04/01/2020 to 15/05/2020. We have collected the fact-checked articles from 92 different fact-checking websites after obtaining references from Poynter and Snopes. We have manually annotated articles into 11 different categories of the fact-checked news according to their content. The dataset is in 40 languages from 105 countries. We have built a classifier to detect fake news and present results for the automatic fake news detection and its class. Our model achieves an F1 score of 0.76 to detect the false class and other fact check articles. The FakeCovid dataset is available at Github.

Summary of factual verification datasets with natural inputs. KG denotes knowledge graphs. ChectThat18 has been extended later {{cite:35a99b8}}, {{cite:d142b1b}}, {{cite:b6e9a7d}}. NELA has been updated by adding more data from more diverse sources {{cite:2462b3f}}, {{cite:89022e6}}, {{cite:d46fb49}}

Next, we discuss the inputs to factual verification. The most popular type of input to verification is textual claims, which is expected given they are often the output of claim detection. These tend to be sentence-level statements, which is a practice common among fact-checkers in order to include only the context relevant to the claim {{cite:868e77e}}. Many existing efforts {{cite:0e577b4}}, {{cite:066b20a}}, {{cite:1f3fc17}}, {{cite:be19473}} constructed datasets by crawling real-world claims from dedicated websites (e.g. Politifact) due to their availability (see Table {{table:d32a86d5-efd5-43db-a548-f75f6dd14010}} ). Unlike previous work that focus on English, {{cite:eec936f}} collected non-English claims from 25 languages.


A decade survey of content based image retrieval using deep learning

deep learning based methods for metric learning in recent 10 years

A Decade Survey of Content Based Image Retrieval using Deep Learning

Abstract: The content based image retrieval aims to find the similar images from a large scale dataset against a query image. Generally, the similarity between the representative features of the query image and dataset images is used to rank the images for retrieval. In early days, various hand designed feature descriptors have been investigated based on the visual cues such as color, texture, shape, etc. that represent the images. However, the deep learning has emerged as a dominating alternative of hand-designed feature engineering from a decade. It learns the features automatically from the data. This paper presents a comprehensive survey of deep learning based developments in the past decade for content based image retrieval. The categorization of existing state-of-the-art methods from different perspectives is also performed for greater understanding of the progress. The taxonomy used in this survey covers different supervision, different networks, different descriptor type and different retrieval type. A performance analysis is also performed using the state-of-the-art methods. The insights are also presented for the benefit of the researchers to observe the progress and to make the best choices. The survey presented in this paper will help in further research progress in image retrieval using deep learning.

A Metric Learning Reality Check

Abstract: Deep metric learning papers from the past four years have consistently claimed great advances in accuracy, often more than doubling the performance of decade-old methods. In this paper, we take a closer look at the field to see if this is actually true. We find flaws in the experimental methodology of numerous metric learning papers, and show that the actual improvements over time have been marginal at best.

Summary of surveys on metric learning.

Scope of our work. The central theme of our paper is supervision,
which we believe is the key to semantic gap resolution. However,
supervision is a broad topic and we need to limit our scope to two
important problems (i.e., object detection and metric learning) as they
help reveal our insights. There are still too many papers on these two
topics, and compiling an extensive list of the state-of-the-art methods
of both is beyond the scope of a paper of reasonable length. Instead,
for domain-specific surveys, readers are referred to
Table {{table:e8335ff4-343c-4feb-b8b6-2cac8dbfee2b}}  and Table {{table:4ba314d1-f1eb-4bf0-a917-fd393299b204}} ,
respectively.


Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Abstract: In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.

Natural Adversarial Examples

Abstract: We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets' real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called ImageNet-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called ImageNet-O, which is the first out-of-distribution detection dataset created for ImageNet models. On ImageNet-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on ImageNet-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

Abstract: We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pre-trained with 1000× more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness.

Microsoft COCO: Common Objects in Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.

Details of the used downstream datasets.

To exhaustively evaluate the proposed Pro-tuning, we perform detailed experiments on fifteen long-standing benchmarks. Details of the used visual datasets are listed in Table {{table:b6fed91f-b48e-4290-acd7-622a7733457e}} . For abbreviation in Table {{table:09e5c675-b45b-4cb1-a03a-ae455bf7a1ca}}  and Table {{table:0c534973-200b-44af-b305-44c8c24ec5b4}} , “C-10-LT”: long-tailed CIFAR-10, “C-100-LT”: long-tailed CIFAR-100, “C-10”: CIFAR-10, “C-100”: CIFAR-100, “PET”: Oxford-IIIT Pets, “Flower”: Oxford Flowers-102, “DOG”: Stanford Dogs, “C1M”: Clothing1M, “C-101”: Caltech-101We show that the proposed Pro-tuning facilitates a board range of vision tasks under seven advanced pre-trained models, including CNN-based and Transformer-based architectures: ResNet-50 {{cite:ffefad9}}, ResNet-101 {{cite:ffefad9}}, RegNetX-32G {{cite:ff21f9a}}, DeiT-S {{cite:ad5ccd6}}, DeiT-B {{cite:ad5ccd6}}, Swin-S {{cite:83f5ea3}}, Swin-B {{cite:83f5ea3}}. During training, we first use the pre-trained weights on ImageNet to initialize pre-trained vision models. In Table {{table:09e5c675-b45b-4cb1-a03a-ae455bf7a1ca}}  and Table {{table:0c534973-200b-44af-b305-44c8c24ec5b4}} , we compute the maximum number of trainable parameters to measure the parameter efficiency, denote by {{formula:709bf9bb-d1ed-4600-be9a-e7283f392bcf}} . More experimental details can be referred to in the supplementary material.


CNNs with an encoding algorithm to convert inertial sensory signals to color images.

A combination of RL and deep networks with different architectures, such as RNN and CNN.

Iss2Image: A Novel Signal-Encoding Technique for CNN-Based Human Activity Recognition

Abstract: The most significant barrier to success in human activity recognition is extracting and selecting the right features. In traditional methods, the features are chosen by humans, which requires the user to have expert knowledge or to do a large amount of empirical study. Newly developed deep learning technology can automatically extract and select features. Among the various deep learning methods, convolutional neural networks (CNNs) have the advantages of local dependency and scale invariance and are suitable for temporal data such as accelerometer (ACC) signals. In this paper, we propose an efficient human activity recognition method, namely Iss2Image (Inertial sensor signal to Image), a novel encoding technique for transforming an inertial sensor signal into an image with minimum distortion and a CNN model for image-based activity classification. Iss2Image converts real number values from the X, Y, and Z axes into three color channels to precisely infer correlations among successive sensor signal values in three different dimensions. We experimentally evaluated our method using several well-known datasets and our own dataset collected from a smartphone and smartwatch. The proposed method shows higher accuracy than other state-of-the-art approaches on the tested datasets.

Creating Pro-Level AI for a Real-Time Fighting Game Using Deep Reinforcement Learning

Abstract: Reinforcement learning (RL) combined with deep neural networks has performed remarkably well in many genres of games recently. It has surpassed human-level performance in fixed game environments and turn-based two-player board games. However, to the best of our knowledge, current research has yet to produce a result that has surpassed human-level performance in modern complex fighting games. This is due to the inherent difficulties with real-time fighting games, including: vast action spaces, action dependencies, and imperfect information. We overcame these challenges and made 1v1 battle AI agents for the commercial game Blade and Soul. The trained agents competed against five professional gamers and achieved a winning rate of 62%. This article presents a practical RL method that includes a novel self-play curriculum and data skipping techniques. Through the curriculum, three different styles of agents were created by reward shaping and were trained against each other. Additionally, this article suggests data-skipping techniques that could increase data efficiency and facilitate explorations in vast spaces. Since our method can be generally applied to all two-player competitive games with vast action spaces, we anticipate its application to game development including level design and automated balancing.

Summary of AI For The Metaverse in The Application Aspect.

In real-time strategy (RTS) games, such as StarCraft, Bayesian models have been used for modeling multi-scale uncertainty and multi-level abstraction levels {{cite:5bf16c4}}: micromanagement, tactics, and strategy. These probabilistic learning models were able to cope reactive units control, recognize objectives from tactical data, and predict opponent’s gameplay based on strategic information.
To reach an intelligent human-like response mechanism, several game software companies has applied AI in various testing tasks during design and development stages. In {{cite:be0f44e}}, a metamorphic testing mechanism was proposed to overcome the impracticability of controlling a large amount of possible moving strategies in artificial chess games. This testing mechanism deployed a decision tree model to reveal metamorphic relations, which in turn effectively determine the optimal move among all possible ones.
With a combination of RL and deep networks, AI agents in {{cite:1e509fb}} were developed to address some inherent difficulties in real-time fighting games and to defeat pro players in one-vs-one battle. In addition to creating different fighting styles through self-play curriculum, this deep RL framework was capable of all two-player competitive games which have level upgrade and balance policies.
RL and supervised learning were also exploited to improve AI agents in RTS games {{cite:0373f0b}}. Being more superior than the Puppet search algorithm, CNNs and deep Q-learning networks inferred the outcome of costly high-level search and optimized the available time to execute tactical searches.
In a nutshell, AI with traditional ML and innovative DL algorithms have been making an unprecedented revolution of gaming experience in many aspects: improving the intelligence of NPCs, modeling complex systems, making games more beautiful and rational, conducting more realistic human-NPC interactions, reducing cost of in-game world creation, and opening more opportunities of developing mobile games.
In Table {{table:dbc4299c-0555-447d-aa57-329848d3b884}} , we summarized the existing application-oriented works utilizing AI technology which have shown the potential to be integrated and deployed in the metaverse.


Quad-networks: unsupervised learning to rank for interest point detection

Abstract: Several machine learning tasks require to represent the data using only a sparse set of interest points. An ideal detector is able to find the corresponding interest points even if the data undergo a transformation typical for a given domain. Since the task is of high practical interest in computer vision, many hand-crafted solutions were proposed. In this paper, we ask a fundamental question: can we learn such detectors from scratch? Since it is often unclear what points are "interesting", human labelling cannot be used to find a truly unbiased solution. Therefore, the task requires an unsupervised formulation. We are the first to propose such a formulation: training a neural network to rank points in a transformation-invariant manner. Interest points are then extracted from the top/bottom quantiles of this ranking. We validate our approach on two tasks: standard RGB image interest point detection and challenging cross-modal interest point detection between RGB and depth images. We quantitatively show that our unsupervised method performs better or on-par with baselines.

LIFT: Learned Invariant Feature Transform

Abstract: We introduce a novel Deep Network architecture that implements the full feature point handling pipeline, that is, detection, orientation estimation, and feature description. While previous works have successfully tackled each one of these problems individually, we show how to learn to do all three in a unified manner while preserving end-to-end differentiability. We then demonstrate that our Deep pipeline outperforms state-of-the-art methods on a number of benchmark datasets, without the need of retraining.

The combinations of detector, orientation estimator and descriptor used.

All the experimental setup was implemented in C++ and Python. The OpenCV 3 implementations were used for all conventional features detectors. The parameters of each method were set to values as close as possible to those given in the original papers, and otherwise to the standard values present in the library. The combinations of detectors and descriptors used for the experiments can be seen in Table {{table:080cd8d6-7edb-4459-9985-30e9ff520b9a}} .


Leveraging Code Generation to Improve Code Retrieval and Summarization via Dual Learning

Abstract: Code summarization generates brief natural language description given a source code snippet, while code retrieval fetches relevant source code given a natural language query. Since both tasks aim to model the association between natural language and programming language, recent studies have combined these two tasks to improve their performance. However, researchers have yet been able to effectively leverage the intrinsic connection between the two tasks as they train these tasks in a separate or pipeline manner, which means their performance can not be well balanced. In this paper, we propose a novel end-to-end model for the two tasks by introducing an additional code generation task. More specifically, we explicitly exploit the probabilistic correlation between code summarization and code generation with dual learning, and utilize the two encoders for code summarization and code generation to train the code retrieval task via multi-task learning. We have carried out extensive experiments on an existing dataset of SQL and Python, and results show that our model can significantly improve the results of the code retrieval task over the-state-of-art models, as well as achieve competitive performance in terms of BLEU score for the code summarization task.

CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning

Abstract: To accelerate software development, much research has been performed to help people understand and reuse the huge amount of available code resources. Two important tasks have been widely studied: code retrieval, which aims to retrieve code snippets relevant to a given natural language query from a code base, and code annotation, where the goal is to annotate a code snippet with a natural language description. Despite their advancement in recent years, the two tasks are mostly explored separately. In this work, we investigate a novel perspective of Code annotation for Code retrieval (hence called “CoaCor”), where a code annotation model is trained to generate a natural language annotation that can represent the semantic meaning of a given code snippet and can be leveraged by a code retrieval model to better distinguish relevant code snippets from others. To this end, we propose an effective framework based on reinforcement learning, which explicitly encourages the code annotation model to generate annotations that can be used for the retrieval task. Through extensive experiments, we show that code annotations generated by our framework are much more detailed and more useful for code retrieval, and they can further improve the performance of existing code retrieval models significantly.1

Multi-modal Attention Network Learning for Semantic Source Code Retrieval

Abstract: Code retrieval techniques and tools have been playing a key role in facilitating software developers to retrieve existing code fragments from available open-source repositories given a user query (e.g., a short natural language text describing the functionality for retrieving a particular code snippet). Despite the existing efforts in improving the effectiveness of code retrieval, there are still two main issues hindering them from being used to accurately retrieve satisfiable code fragments from large-scale repositories when answering complicated queries. First, the existing approaches only consider shallow features of source code such as method names and code tokens, but ignoring structured features such as abstract syntax trees (ASTs) and control-flow graphs (CFGs) of source code, which contains rich and well-defined semantics of source code. Second, although the deep learning-based approach performs well on the representation of source code, it lacks the explainability, making it hard to interpret the retrieval results and almost impossible to understand which features of source code contribute more to the final results. To tackle the two aforementioned issues, this paper proposes MMAN, a novel Multi-Modal Attention Network for semantic source code retrieval. A comprehensive multi-modal representation is developed for representing unstructured and structured features of source code, with one LSTM for the sequential tokens of code, a Tree-LSTM for the AST of code and a GGNN (Gated Graph Neural Network) for the CFG of code. Furthermore, a multi-modal attention fusion layer is applied to assign weights to different parts of each modality of source code and then integrate them into a single hybrid representation. Comprehensive experiments and analysis on a large-scale real-world dataset show that our proposed model can accurately retrieve code snippets and outperforms the state-of-the-art methods.

When deep learning met code search

Abstract: There have been multiple recent proposals on using deep neural networks for code search using natural language. Common across these proposals is the idea of embedding code and natural language queries into real vectors and then using vector distance to approximate semantic correlation between code and the query. Multiple approaches exist for learning these embeddings, including unsupervised techniques, which rely only on a corpus of code examples, and supervised techniques, which use an aligned corpus of paired code and natural language descriptions. The goal of this supervision is to produce embeddings that are more similar for a query and the corresponding desired code snippet. Clearly, there are choices in whether to use supervised techniques at all, and if one does, what sort of network and training to use for supervision. This paper is the first to evaluate these choices systematically. To this end, we assembled implementations of state-of-the-art techniques to run on a common platform, training and evaluation corpora. To explore the design space in network complexity, we also introduced a new design point that is a minimal supervision extension to an existing unsupervised technique. Our evaluation shows that: 1. adding supervision to an existing unsupervised technique can improve performance, though not necessarily by much; 2. simple networks for supervision can be more effective that more sophisticated sequence-based networks for code search; 3. while it is common to use docstrings to carry out supervision, there is a sizeable gap between the effectiveness of docstrings and a more query-appropriate supervision corpus.

Query Expansion Based on Crowd Knowledge for Code Search

Abstract: As code search is a frequent developer activity in software development practices, improving the performance of code search is a critical task. In the text retrieval based search techniques employed in the code search, the term mismatch problem is a critical language issue for retrieval effectiveness. By reformulating the queries, query expansion provides effective ways to solve the term mismatch problem. In this paper, we propose Query Expansion based on Crowd Knowledge (QECK), a novel technique to improve the performance of code search algorithms. QECK identifies software-specific expansion words from the high quality pseudo relevance feedback question and answer pairs on Stack Overflow to automatically generate the expansion queries. Furthermore, we incorporate QECK in the classic Rocchio's model, and propose QECK based code search method QECKRocchio. We conduct three experiments to evaluate our QECK technique and investigate QECKRocchio in a large-scale corpus containing real-world code snippets and a question and answer pair collection. The results show that QECK improves the performance of three code search algorithms by up to 64 percent in Precision, and 35 percent in NDCG. Meanwhile, compared with the state-of-the-art query expansion method, the improvement of QECK Rocchio is 22 percent in Precision, and 16 percent in NDCG.

ROSF: Leveraging Information Retrieval and Supervised Learning for Recommending Code Snippets

Abstract: When implementing unfamiliar programming tasks, developers commonly search code examples and learn usage patterns of APIs from the code examples or reuse them by copy-pasting and modifying. For providing high-quality code examples, previous studies present several methods to recommend code snippets mainly based on information retrieval. In this paper, to provide better recommendation results, we propose ROSF, Recommending cOde Snippets with multi-aspect Features, a novel method combining both information retrieval and supervised learning. In our method, we recommend Top-Kcode snippets for a givenfree-form query based on two stages, i.e., coarse-grained searching and fine-grained re-ranking. First, we generate a code snippet candidate set by searching a code snippet corpus using an information retrieval method. Second, we predict probability values of the code snippets for different relevance scores in the candidate set by the learned prediction model from a training set, re-rank these candidate code snippets according to the probability values, and recommend the final results to developers. We conduct several experiments to evaluate our method in a large-scale corpus containing 921,713 real-world code snippets. The results show that ROSF is an effective method for code snippets recommendation and outperforms the-state-of-the-art methods by 20%-41% in Precision and 13%-33% in NDCG

Text-based code search tools.

The goal of text-based code search is to retrieve source code from a large-scale code corpus that most closely matches the free-form input by developers {{cite:14affd7}}, {{cite:b285d6d}}. Reusing existing code can largely boost developers' coding efficiency and potentially also quality, by reusing high quality code examples. Text-based code search is frequently studied because this task aims to improve the performance of frequently used code search engines in practice, such as GitHub search. Table {{table:54f56187-d37a-4b3c-a88c-2b4bc6f0c104}}  presents text-based code search tools with their major and auxiliary modeling techniques. This table also shows how different tools are related to each other in terms of their compared baseline tools.


[tl]full-time computer science researchers at Microsoft Research

1 doctoral supervisor, 2 master supervisors, 2 graduate students

Modeling User Behaviour in Research Paper Recommendation System

Abstract: User intention which often changes dynamically is considered to be an important factor for modeling users in the design of recommendation systems. Recent studies are starting to focus on predicting user intention (what users want) beyond user preference (what users like). In this work, a user intention model is proposed based on deep sequential topic analysis. The model predicts a user's intention in terms of the topic of interest. The Hybrid Topic Model (HTM) comprising Latent Dirichlet Allocation (LDA) and Word2Vec is proposed to derive the topic of interest of users and the history of preferences. HTM finds the true topics of papers estimating word-topic distribution which includes syntactic and semantic correlations among words. Next, to model user intention, a Long Short Term Memory (LSTM) based sequential deep learning model is proposed. This model takes into account temporal context, namely the time difference between clicks of two consecutive papers seen by a user. Extensive experiments with the real-world research paper dataset indicate that the proposed approach significantly outperforms the state-of-the-art methods. Further, the proposed approach introduces a new road map to model a user activity suitable for the design of a research paper recommendation system.

A Scalable Hybrid Research Paper Recommender System for Microsoft Academic

Abstract: We present the design and methodology for the large scale hybrid paper recommender system used by Microsoft Academic. The system provides recommendations for approximately 160 million English research papers and patents. Our approach handles incomplete citation information while also alleviating the cold-start problem that often affects other recommender systems. We use the Microsoft Academic Graph (MAG), titles, and available abstracts of research papers to build a recommendation list for all documents, thereby combining co-citation and content based approaches. Tuning system parameters also allows for blending and prioritization of each approach which, in turn, allows us to balance paper novelty versus authority in recommendation results. We evaluate the generated recommendations via a user study of 40 participants, with over 2400 recommendation pairs graded and discuss the quality of the results using P@10 and nDCG scores. We see that there is a strong correlation between participant scores and the similarity rankings produced by our system but that additional focus needs to be put towards improving recommender precision, particularly for content based recommendations. The results of the user survey and associated analysis scripts are made available via GitHub and the recommendations produced by our system are available as part of the MAG on Azure to facilitate further research and light up novel research paper recommendation applications.

In-text citation’s frequencies-based recommendations of relevant research papers

Abstract: From the past half of a century, identification of the relevant documents is deemed an active area of research due to the rapid increase of data on the web. The traditional models to retrieve relevant documents are based on bibliographic information such as Bibliographic coupling, Co-citations, and Direct citations. However, in the recent past, the scientific community has started to employ textual features to improve existing models’ accuracy. In our previous study, we found that analysis of citations at a deep level (i.e., content level) can play a paramount role in finding more relevant documents than surface level (i.e., just bibliography details). We found that cited and citing papers have a high degree of relevancy when in-text citations frequency of the cited paper is more than five times in the citing paper’s text. This paper is an extension of our previous study in terms of its evaluation of a comprehensive dataset. Moreover, the study results are also compared with other state-of-the-art approaches i.e., content, metadata, and bibliography. For evaluation, a user study is conducted on selected papers from 1,200 documents (comprise about 16,000 references) of an online journal, Journal of Computer Science (J.UCS). The evaluation results indicate that in-text citation frequency has attained higher precision in finding relevant papers than other state-of-the-art techniques such as content, bibliographic coupling, and metadata-based techniques. The use of in-text citation may help in enhancing the quality of existing information systems and digital libraries. Further, more sophisticated measure may be redefined be considering the use of in-text citations.

A Novel Paper Recommendation Method Empowered by Knowledge Graph: for Research Beginners

Abstract: Searching for papers from different academic databases is the most commonly used method by research beginners to obtain cross-domain technical solutions. However, it is usually inefficient and sometimes even useless because traditional search methods neither consider knowledge heterogeneity in different domains nor build the bottom layer of search, including but not limited to the characteristic description text of target solutions and solutions to be excluded. To alleviate this problem, a novel paper recommendation method is proposed herein by introducing "master-slave" domain knowledge graphs, which not only help users express their requirements more accurately but also helps the recommendation system better express knowledge. Specifically, it is not restricted by the cold start problem and is a challenge-oriented method. To identify the rationality and usefulness of the proposed method, we selected two cross-domains and three different academic databases for verification. The experimental results demonstrate the feasibility of obtaining new technical papers in the cross-domain scenario by research beginners using the proposed method. Further, a new research paradigm for research beginners in the early stages is proposed herein.

For all observed works with user studies we list their number of participants (#P) and their composition. NA indicates that #P or compositions were not described in a specific user study.

A distinction in lab-based vs. real world user studies can be conducted {{cite:39840b8}}, {{cite:490b097}}. User studies where participants rate recommendations according to some criteria and are aware of the study are lab-based, all others are considered real world studies. Living labs {{cite:67a79f8}}, {{cite:b4c5250}}, {{cite:e1e105a}} for example enable real world user studies.
On average the lab-based user studies were conducted with 17.83 users. Table {{table:522edd8c-1b76-4146-8b88-1e62d4476207}}  holds information on the number of participants for all studies as well as the composition of groups in terms of seniority.

Paper recommendation is always targeted at human users. But oftentimes an evaluation with real users to quantify users' satisfaction with recommended publications is simply not conducted {{cite:733c2c6}}. Conducting huge user  studies  is  not  feasible {{cite:93ea9d1}}.  So  sometimes  user data  to  evaluate  with  is  fetched  from  the  presented datasets {{cite:1b6f991}}, {{cite:640a003}} or user behaviour is artificially emulated {{cite:8a36b3e}}, {{cite:14eff14}}, {{cite:15521f7}}.
Noteworthy counter-examplesFor a full list of approaches conducting user studies see Table {{table:522edd8c-1b76-4146-8b88-1e62d4476207}} . are the studies by Bulut et al. {{cite:26f79f1}} who emailed 50 researchers to rate relevancy of recommended articles or Chaudhuri et al. {{cite:da1dd65}} who asked 45 participants to rate their acceptance of recommended publications.
Another option to overcome this issue is utilisation of living labs as seen with ArXivDigest {{cite:67a79f8}}, Mr. DLib's living lab {{cite:b4c5250}} or LiLAS for the related tasks of dataset recommendation for scientific publications and multi-lingual document retrieval {{cite:e1e105a}}.

User characteristics such as registration status of users are already mentioned by Beel et al. {{cite:39840b8}} as a factor which is disregarded in evaluations. We want to extend on this point and highlight the oftentimes missing or inadequate descriptions of intended users of paper recommendation systems. Traits of users and their information needs are not only important for experiments but should also be regarded in the construction of an approach. The targeted audience of a paper recommendation system should influence its suggestions. Bai et al. {{cite:82f1b32}} highlight different needs of junior researchers which should be recommended a broad variety of papers as they still have to figure out their direction. They state recommendations for senior researchers should be more in line with their already established interests. Sugiyama and Kan {{cite:2606fba}} describe the need to help discover interdisciplinary research for this experienced user group.
Most works do not recognise possible different functions of paper recommendation systems for users depending on their level of seniority. If papers include an evaluation with real persons, they e.g. mix Master's students with professors but do not address their different goals or expectations from paper recommendation {{cite:0123a43}}. Chaudhuri et al. {{cite:da1dd65}} have junior, experienced and expert users as participants of their study and give individual ratings but do not calculate evaluation scores per user group.
In some studies the exact composition of test users is not even mentioned (see Table {{table:522edd8c-1b76-4146-8b88-1e62d4476207}} ).


Optimization under Decision-Dependent Uncertainty

Abstract: The efficacy of robust optimization spans a variety of settings with uncertainties bounded in predetermined sets. In many applications, uncertainties are affected by decisions and cannot be modeled with current frameworks. This paper takes a step towards generalizing robust linear optimization to problems with decision-dependent uncertainties. In general settings, we show these problems to be NP-complete. To alleviate the computational inefficiencies, we introduce a class of uncertainty sets whose size depends on binary decisions. We propose reformulations that improve upon alternative standard linearization techniques. To illustrate the advantages of this framework, a shortest path problem is discussed, where the uncertain arc lengths are affected by decisions. Beyond the modeling and performance advantages, the proposed notion of proactive uncertainty control also mitigates over conservatism of current robust optimization approaches.

Robust Scheduling of Virtual Power Plant Under Exogenous and Endogenous Uncertainties

Abstract: Virtual power plant (VPP) provides a flexible solution to distributed energy resources integration by aggregating renewable generation units, conventional power plants, energy storages, and flexible demands. This paper proposes a novel stochastic adaptive robust optimization (SARO) model for determining the optimal self-scheduling plan for VPP’s participation in the day-ahead energy-reserve market. We consider exogenous uncertainties (or called decision-independent uncertainties, DIUs) associated with market clearing prices and available wind generation, as well as endogenous uncertainties (or called decision-dependent uncertainties, DDUs) pertaining to real-time reserve deployment requests. A tractable solution methodology based on modified Benders dual decomposition is developed to effectively solve the proposed SARO model with both DIUs and DDUs. Case studies are conducted to verify the efficiency and applicability of the proposed approach. Comparative results show that the proposed method can mitigate the conservatism of robust strategy by capturing a satisfactory trade-off between profitability and real-time operation feasibility.

A Survey of User Perspectives on Security and Privacy in a Home Networking Environment

Abstract: The security and privacy of smart home systems, particularly from a home user's perspective, have been a very active research area in recent years. However, via a meta-review of 52 review papers covering related topics (published between 2000 and 2021), this paper shows a lack of a more recent literature review on user perspectives of smart home security and privacy since the 2010s. This identified gap motivated us to conduct a systematic literature review (SLR) covering 126 relevant research papers published from 2010 to 2021. Our SLR led to the discovery of a number of important areas where further research is needed; these include holistic methods that consider a more diverse and heterogeneous range of home devices, interactions between multiple home users, complicated data flow between multiple home devices and home users, some less-studied demographic factors, and advanced conceptual frameworks. Based on these findings, we recommended key future research directions, e.g., research for a better understanding of security and privacy aspects in different multi-device and multi-user contexts, and a more comprehensive ontology on the security and privacy of the smart home covering varying types of home devices and behaviors of different types of home users.

Summary of recent work on DDU

When the curtailment of renewable energy is incorporated in the pre-dispatch stage, the variation range of real-time RG output will be influenced. Therefore, distinct from the uncertainty sets in traditional robust optimization models that are pre-specified, the uncertainty set in our model depends on the first-stage decision. Decision-dependent uncertainty (DDU) has captured great attention in recent years and mainly two types of DDU have been considered {{cite:b98046a}}: 1) information structure related DDU (Type-1), i.e., decisions affect when knowledge about uncertain factors becomes available; 2) uncertainty distribution related DDU (Type-2), i.e., decisions affect the size of an uncertainty set or the chance that a scenario happens. Related studies are summarized in TABLE {{table:c54a92eb-3660-4218-8c43-effcdb75c698}} . Stochastic programming was applied to deal with DDU {{cite:95af7eb}}, {{cite:3f4433a}}, {{cite:cc58f39}}. As for robust optimization, reference {{cite:20a8b55}} provided a reformulation method to solve the static robust model with DDU in which only “here-and-now” variables were included and the size of uncertainty set merely depends on binary variables. Another robust counterpart model was established in {{cite:d2b76a1}} with a more general set. In practice, adjustable robust optimization is more commonly used since it can describe the adjustment behaviors after the realization of uncertain factors. The solution of an adjustable robust optimization with DDU can be estimated via a K-adaptability approximation based algorithm {{cite:c2ba4a4}}. Multi-parametric programming {{cite:650913f}}, modified Bender’s decomposition {{cite:5ee3fce}}, {{cite:612e154}}, and improved C&CG {{cite:fc60063}} were deployed to offer an exact solution. However, they are either time-consuming with a growing number of units or concentrated on the linear DDU set. In general, robust optimization with DDU is an important albeit challenging problem due to its computational intractability {{cite:b437527}}.