Multi-Channel Graph Neural Network for Entity Alignment

Abstract: Entity alignment typically suffers from the issues of structural heterogeneity and limited seed alignments. In this paper, we propose a novel Multi-channel Graph Neural Network model (MuGNN) to learn alignment-oriented knowledge graph (KG) embeddings by robustly encoding two KGs via multiple channels. Each channel encodes KGs via different relation weighting schemes with respect to self-attention towards KG completion and cross-KG attention for pruning exclusive entities respectively, which are further combined via pooling techniques. Moreover, we also infer and transfer rule knowledge for completing two KGs consistently. MuGNN is expected to reconcile the structural differences of two KGs, and thus make better use of seed alignments. Extensive experiments on five publicly available datasets demonstrate our superior performance (5% Hits@1 up on average).

Cross-lingual Entity Alignment via Joint Attribute-Preserving Embedding

Abstract: Entity alignment is the task of finding entities in two knowledge bases (KBs) that represent the same real-world object. When facing KBs in different natural languages, conventional cross-lingual entity alignment methods rely on machine translation to eliminate the language barriers. These approaches often suffer from the uneven quality of translations between languages. While recent embedding-based techniques encode entities and relationships in KBs and do not need machine translation for cross-lingual entity alignment, a significant number of attributes remain largely unexplored. In this paper, we propose a joint attribute-preserving embedding model for cross-lingual entity alignment. It jointly embeds the structures of two KBs into a unified vector space and further refines it by leveraging attribute correlations in the KBs. Our experimental results on real-world datasets show that this approach significantly outperforms the state-of-the-art embedding approaches for cross-lingual entity alignment and could be complemented with methods based on machine translation.

Knowledge Association with Hyperbolic Knowledge Graph Embeddings

Abstract: Capturing associations for knowledge graphs (KGs) through entity alignment, entity type inference and other related tasks benefits NLP applications with comprehensive knowledge representations. Recent related methods built on Euclidean embeddings are challenged by the hierarchical structures and different scales of KGs. They also depend on high embedding dimensions to realize enough expressiveness. Differently, we explore with low-dimensional hyperbolic embeddings for knowledge association. We propose a hyperbolic relational graph neural network for KG embedding and capture knowledge associations with a hyperbolic transformation. Extensive experiments on entity alignment and type inference demonstrate the effectiveness and efficiency of our method.

TransEdge: Translating Relation-contextualized Embeddings for Knowledge Graphs

Abstract: Learning knowledge graph (KG) embeddings has received increasing attention in recent years. Most embedding models in literature interpret relations as linear or bilinear mapping functions to operate on entity embeddings. However, we find that such relation-level modeling cannot capture the diverse relational structures of KGs well. In this paper, we propose a novel edge-centric embedding model TransEdge, which contextualizes relation representations in terms of specific head-tail entity pairs. We refer to such contextualized representations of a relation as edge embeddings and interpret them as translations between entity embeddings. TransEdge achieves promising performance on different prediction tasks. Our experiments on benchmark datasets indicate that it obtains the state-of-the-art results on embedding-based entity alignment. We also show that TransEdge is complementary with conventional entity alignment methods. Moreover, it shows very competitive performance on link prediction.

Cross-lingual Knowledge Graph Alignment via Graph Matching Neural Network

Abstract: Previous cross-lingual knowledge graph (KG) alignment studies rely on entity embeddings derived only from monolingual KG structural information, which may fail at matching entities that have different facts in two KGs. In this paper, we introduce the topic entity graph, a local sub-graph of an entity, to represent entities with their contextual information in KG. From this view, the KB-alignment task can be formulated as a graph matching problem; and we further propose a graph-attention based solution, which first matches all entities in two topic entity graphs, and then jointly model the local matching information to derive a graph-level matching vector. Experiments show that our model outperforms previous state-of-the-art methods by a large margin.

Relation-Aware Entity Alignment for Heterogeneous Knowledge Graphs

Abstract: Entity alignment is the task of linking entities with the same real-world identity from different knowledge graphs (KGs), which has been recently dominated by embedding-based methods. Such approaches work by learning KG representations so that entity alignment can be performed by measuring the similarities between entity embeddings. While promising, prior works in the field often fail to properly capture complex relation information that commonly exists in multi-relational KGs, leaving much room for improvement. In this paper, we propose a novel Relation-aware Dual-Graph Convolutional Network (RDGCN) to incorporate relation information via attentive interactions between the knowledge graph and its dual relation counterpart, and further capture neighboring structures to learn better entity representations. Experiments on three real-world cross-lingual datasets show that our approach delivers better and more robust results over the state-of-the-art alignment methods by learning better KG representations.

Aligning Cross-Lingual Entities with Multi-Aspect Information

Abstract: Multilingual knowledge graphs (KGs), such as YAGO and DBpedia, represent entities in different languages. The task of cross-lingual entity alignment is to match entities in a source language with their counterparts in target languages. In this work, we investigate embedding-based approaches to encode entities from multilingual KGs into the same vector space, where equivalent entities are close to each other. Specifically, we apply graph convolutional networks (GCNs) to combine multi-aspect information of entities, including topological connections, relations, and attributes of entities, to learn entity embeddings. To exploit the literal descriptions of entities expressed in different languages, we propose two uses of a pretrained multilingual BERT model to bridge cross-lingual gaps. We further propose two strategies to integrate GCN-based and BERT-based modules to boost performance. Extensive experiments on two benchmark datasets demonstrate that our method significantly outperforms existing systems.

Deep Graph Matching Consensus

Abstract: This work presents a two-stage neural architecture for learning and refining structural correspondences between graphs. First, we use localized node embeddings computed by a graph neural network to obtain an initial ranking of soft correspondences between nodes. Secondly, we employ synchronous message passing networks to iteratively re-rank the soft correspondences to reach a matching consensus in local neighborhoods between graphs. We show, theoretically and empirically, that our message passing scheme computes a well-founded measure of consensus for corresponding neighborhoods, which is then used to guide the iterative re-ranking process. Our purely local and sparsity-aware architecture scales well to large, real-world inputs while still being able to recover global correspondences consistently. We demonstrate the practical effectiveness of our method on real-world tasks from the fields of computer vision and entity alignment between knowledge graphs, on which we improve upon the current state-of-the-art.

Categorization of some popular EA methods.

Most of existing EA methods can be summarized into two steps:
(1) Using KG embedding methods (e.g., TransE {{cite:e05317a}}, GCN {{cite:3b669aa}}, and GAT {{cite:c432a2e}}) to generate low-dimensional embeddings for entities and relations in each KG.
(2) Mapping these embeddings into a unified vector space through pre-aligned entities and pairing each entity by distance metrics (e.g., Cosine and Manhattan).
In addition, several methods apply iterative strategies {{cite:d6d88ec}}, {{cite:d1a1d75}} to generate semi-supervised data or introduce extra literal information {{cite:2abdb26}}, {{cite:2ead7f3}}, {{cite:e03d469}} (e.g., entity name) to enhance the model.
In Table {{table:069cad50-af2e-4f14-baa0-40dddac14e32}} , EA methods are categorized based on what design is chosen for encoder and mapper and whether introducing the enhancement module.


Multilingual Knowledge Graph Embeddings for Cross-lingual Knowledge Alignment

Abstract: Many recent works have demonstrated the benefits of knowledge graph embeddings in completing monolingual knowledge graphs. Inasmuch as related knowledge bases are built in several different languages, achieving cross-lingual knowledge alignment will help people in constructing a coherent knowledge base, and assist machines in dealing with different expressions of entity relationships across diverse human languages. Unfortunately, achieving this highly desirable crosslingual alignment by human labor is very costly and errorprone. Thus, we propose MTransE, a translation-based model for multilingual knowledge graph embeddings, to provide a simple and automated solution. By encoding entities and relations of each language in a separated embedding space, MTransE provides transitions for each embedding vector to its cross-lingual counterparts in other spaces, while preserving the functionalities of monolingual embeddings. We deploy three different techniques to represent cross-lingual transitions, namely axis calibration, translation vectors, and linear transformations, and derive five variants for MTransE using different loss functions. Our models can be trained on partially aligned graphs, where just a small portion of triples are aligned with their cross-lingual counterparts. The experiments on cross-lingual entity matching and triple-wise alignment verification show promising results, with some variants consistently outperforming others on different tasks. We also explore how MTransE preserves the key properties of its monolingual counterpart TransE.

Learning to Exploit Long-term Relational Dependencies in Knowledge Graphs

Abstract: We study the problem of knowledge graph (KG) embedding. A widely-established assumption to this problem is that similar entities are likely to have similar relational roles. However, existing related methods derive KG embeddings mainly based on triple-level learning, which lack the capability of capturing long-term relational dependencies of entities. Moreover, triple-level learning is insufficient for the propagation of semantic information among entities, especially for the case of cross-KG embedding. In this paper, we propose recurrent skipping networks (RSNs), which employ a skipping mechanism to bridge the gaps between entities. RSNs integrate recurrent neural networks (RNNs) with residual learning to efficiently capture the long-term relational dependencies within and between KGs. We design an end-to-end framework to support RSNs on different tasks. Our experimental results showed that RSNs outperformed state-of-the-art embedding-based methods for entity alignment and achieved competitive performance for KG completion.

Knowledge Graph Alignment Network with Gated Multi-hop Neighborhood Aggregation

Abstract: Graph neural networks (GNNs) have emerged as a powerful paradigm for embedding-based entity alignment due to their capability of identifying isomorphic subgraphs. However, in real knowledge graphs (KGs), the counterpart entities usually have non-isomorphic neighborhood structures, which easily causes GNNs to yield different representations for them. To tackle this problem, we propose a new KG alignment network, namely AliNet, aiming at mitigating the non-isomorphism of neighborhood structures in an end-to-end manner. As the direct neighbors of counterpart entities are usually dissimilar due to the schema heterogeneity, AliNet introduces distant neighbors to expand the overlap between their neighborhood structures. It employs an attention mechanism to highlight helpful distant neighbors and reduce noises. Then, it controls the aggregation of both direct and distant neighborhood information using a gating mechanism. We further propose a relation loss to refine entity representations. We perform thorough experiments with detailed ablation studies and analyses on five entity alignment datasets, demonstrating the effectiveness of AliNet.

Jointly Learning Entity and Relation Representations for Entity Alignment

Abstract: Entity alignment is a viable means for integrating heterogeneous knowledge among different knowledge graphs (KGs). Recent developments in the field often take an embedding-based approach to model the structural information of KGs so that entity alignment can be easily performed in the embedding space. However, most existing works do not explicitly utilize useful relation representations to assist in entity alignment, which, as we will show in the paper, is a simple yet effective way for improving entity alignment. This paper presents a novel joint learning framework for entity alignment. At the core of our approach is a Graph Convolutional Network (GCN) based framework for learning both entity and relation representations. Rather than relying on pre-aligned relation seeds to learn relation representations, we first approximate them using entity embeddings learned by the GCN. We then incorporate the relation approximation into entities to iteratively learn better representations for both. Experiments performed on three real-world cross-lingual datasets show that our approach substantially outperforms state-of-the-art entity alignment methods.

Time costs of EA methods (seconds).

Time Efficiency.
The trump card of PSR is superior efficiency and scalability.
So we specifically evaluate the overall time costs of existing EA methods and report all results in Table {{table:80aa937d-7b1b-4d73-a8ce-d1bce8f247c8}} .
It is obvious that the efficiency of PSR far exceeds all advanced competitors.
The speed of our proposed method is tens or even hundreds of times faster than that of existing complex EA methods.
Even compared against the fastest baseline (i.e., GCN-Align), the speed of PSR (basic) is {{formula:3404f1b6-724d-44fb-9164-01eeb6df4324}}  faster on DWY100K, while the {{formula:2258fe94-56cf-487c-9f5e-8533d0fc9023}}  is {{formula:f9dde81b-7cb2-43ae-a493-b8a9aa227b30}}  higher.
Among the PSR group itself, the semi-supervised strategy consumes {{formula:26dca0b5-642e-4a63-91f2-4d8d557f5231}}  more time cost than the basic version, while PSR (lit) has no clear additional time cost than PSR (semi).


Discriminative Predicate Path Mining for Fact Checking in Knowledge Graphs

Abstract: Traditional fact checking by experts and analysts cannot keep pace with the volume of newly created information. It is important and necessary, therefore, to enhance our ability to computationally determine whether some statement of fact is true or false. We view this problem as a link-prediction task in a knowledge graph, and present a discriminative path-based method for fact checking in knowledge graphs that incorporates connectivity, type information, and predicate interactions. Given a statement S of the form (subject, predicate, object), for example, (Chicago, capitalOf, Illinois), our approach mines discriminative paths that alternatively define the generalized statement (U.S. city, predicate, U.S. state) and uses the mined rules to evaluate the veracity of statement S. We evaluate our approach by examining thousands of claims related to history, geography, biology, and politics using a public, million node knowledge graph extracted from Wikipedia and PubMedDB. Not only does our approach significantly outperform related models, we also find that the discriminative predicate path model is easily interpretable and provides sensible reasons for the final determination.

Finding Streams in Knowledge Graphs to Support Fact Checking

Abstract: The volume and velocity of information that gets generated online limits current journalistic practices to fact-check claims at the same rate. Computational approaches for fact checking may be the key to help mitigate the risks of massive misinformation spread. Such approaches can be designed to not only be scalable and effective at assessing veracity of dubious claims, but also to boost a human fact checker's productivity by surfacing relevant facts and patterns to aid their analysis. To this end, we present a novel, unsupervised network-flow based approach to determine the truthfulness of a statement of fact expressed in the form of a (subject, predicate, object) triple. We view a knowledge graph of background information about real-world entities as a flow network, and knowledge as a fluid, abstract commodity. We show that computational fact checking of such a triple then amounts to finding a "knowledge stream" that emanates from the subject node and flows toward the object node through paths connecting them. Evaluation on a range of real-world and hand-crafted datasets of facts related to entertainment, business, sports, geography and more reveals that this network-flow model can be very effective in discerning true statements from false ones, outperforming existing algorithms on many test cases. Moreover, the model is expressive in its ability to automatically discover several useful path patterns and surface relevant facts that may help a human fact checker corroborate or refute a claim.

Automatic Detection of Fake News

Abstract: The proliferation of misleading information in everyday access media outlets such as social media feeds, news blogs, and online newspapers have made it challenging to identify trustworthy news sources, thus increasing the need for computational tools able to provide insights into the reliability of online content. In this paper, we focus on the automatic identification of fake content in online news. Our contribution is twofold. First, we introduce two novel datasets for the task of fake news detection, covering seven different news domains. We describe the collection, annotation, and validation process in detail and present several exploratory analyses on the identification of linguistic differences in fake and legitimate news content. Second, we conduct a set of learning experiments to build accurate fake news detectors, and show that we can achieve accuracies of up to 76%. In addition, we provide comparative analyses of the automatic and manual identification of fake news.

FEVER: a Large-scale Dataset for Fact Extraction and VERification

Abstract: In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from. The claims are classified as Supported, Refuted or NotEnoughInfo by annotators achieving 0.6841 in Fleiss kappa. For the first two classes, the annotators also recorded the sentence(s) forming the necessary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach and compare it to suitably designed oracles. The best accuracy we achieve on labeling a claim accompanied by the correct evidence is 31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources.

HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification

Abstract: We introduce HoVer (HOppy VERification), a dataset for many-hop evidence extraction and fact verification. It challenges models to extract facts from several Wikipedia articles that are relevant to a claim and classify whether the claim is Supported or Not-Supported by the facts. In HoVer, the claims require evidence to be extracted from as many as four English Wikipedia articles and embody reasoning graphs of diverse shapes. Moreover, most of the 3/4-hop claims are written in multiple sentences, which adds to the complexity of understanding long-range dependency relations such as coreference. We show that the performance of an existing state-of-the-art semantic-matching model degrades significantly on our dataset as the number of reasoning hops increases, hence demonstrating the necessity of many-hop reasoning to achieve strong results. We hope that the introduction of this challenging dataset and the accompanying evaluation task will encourage research in many-hop fact retrieval and information verification. We make the HoVer dataset publicly available at https://hover-nlp.github.io

Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence

Abstract: Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with additional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VitaminC are contrastive, i.e., they contain evidence pairs that are nearly identical in language and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness -- improving accuracy by 10% on adversarial fact verification and 6% on adversarial natural language inference (NLI). Moreover, the structure of VitaminC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evidence for verifying the claim, identifying factual revisions, and providing automatic edits via factually consistent text generation.

TabFact: A Large-scale Dataset for Table-based Fact Verification

Abstract: The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains under-explored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far behind human performance. We also perform a comprehensive analysis to demonstrate great future opportunities. The data and code of the dataset are provided in \url{https://github.com/wenhuchen/Table-Fact-Checking}.

INFOTABS: Inference on Tables as Semi-structured Data

Abstract: In this paper, we observe that semi-structured tabulated text is ubiquitous; understanding them requires not only comprehending the meaning of text fragments, but also implicit relationships between them. We argue that such data can prove as a testing ground for understanding how we reason about information. To study this, we introduce a new dataset called INFOTABS, comprising of human-written textual hypotheses based on premises that are tables extracted from Wikipedia info-boxes. Our analysis shows that the semi-structured, multi-domain and heterogeneous nature of the premises admits complex, multi-faceted reasoning. Experiments reveal that, while human annotators agree on the relationships between a table-hypothesis pair, several standard modeling strategies are unsuccessful at the task, suggesting that reasoning about tables can pose a difficult modeling challenge.

SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS)

Abstract: Understanding tables is an important and relevant task that involves understanding table structure as well as being able to compare and contrast information within cells. In this paper, we address this challenge by presenting a new dataset and tasks that addresses this goal in a shared task in SemEval 2020 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS). Our dataset contains 981 manually-generated tables and an auto-generated dataset of 1980 tables providing over 180K statement and over 16M evidence annotations. SEM-TAB-FACTS featured two sub-tasks. In sub-task A, the goal was to determine if a statement is supported, refuted or unknown in relation to a table. In sub-task B, the focus was on identifying the specific cells of a table that provide evidence for the statement. 69 teams signed up to participate in the task with 19 successful submissions to subtask A and 12 successful submissions to subtask B. We present our results and main findings from the competition.

FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information

Abstract: Fact verification has attracted a lot of attention in the machine learning and natural language processing communities, as it is one of the key methods for detecting misinformation. Existing large-scale benchmarks for this task have focused mostly on textual sources, i.e. unstructured information, and thus ignored the wealth of information available in structured formats, such as tables. In this paper we introduce a novel dataset and benchmark, Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS), which consists of 87,026 verified claims. Each claim is annotated with evidence in the form of sentences and/or cells from tables in Wikipedia, as well as a label indicating whether this evidence supports, refutes, or does not provide enough information to reach a verdict. Furthermore, we detail our efforts to track and minimize the biases present in the dataset and could be exploited by models, e.g. being able to predict the label without using evidence. Finally, we develop a baseline for verifying claims against text and tables which predicts both the correct evidence and verdict for 18% of the claims.

Stance Prediction and Claim Verification: An Arabic Perspective

Abstract: This work explores the application of textual entailment in news claim verification and stance prediction using a new corpus in Arabic. The publicly available corpus comes in two perspectives: a version consisting of 4,547 true and false claims and a version consisting of 3,786 pairs (claim, evidence). We describe the methodology for creating the corpus and the annotation process. Using the introduced corpus, we also develop two machine learning baselines for two proposed tasks: claim verification and stance prediction. Our best model utilizes pretraining (BERT) and achieves 76.7 F1 on the stance prediction task and 64.3 F1 on the claim verification task. Our preliminary experiments shed some light on the limits of automatic claim verification that relies on claims text only. Results hint that while the linguistic features and world knowledge learned during pretraining are useful for stance prediction, such learned representations from pretraining are insufficient for verifying claims without access to context or evidence.

Summary of factual verification datasets with artificial inputs. Google denotes Google Relation Extraction Corpora, and WSDM means Cup Triple Scoring Challenge.

In order to better control the complexity of the task, efforts listed in Table {{table:7e0cdaba-af8c-4bbc-90b8-42f57e738948}}  created claims artificially. {{cite:cfaac38}} had annotators mutate sentences from Wikipedia articles to create claims. Following the same approach, {{cite:8b6c474}} and {{cite:920e135}} constructed Arabic and Danish datasets respectively. Another frequently considered option is subject-predicate-object triples, e.g. (London, city_in, UK). The popularity of triples as input stems from the fact that they facilitate fact-checking against knowledge bases {{cite:073c151}}, {{cite:3f13175}}, {{cite:a74363b}}, {{cite:946ddc4}} such as DBpedia {{cite:9f182de}}, SemMedDB {{cite:26cc535}}, and KBox {{cite:a51c375}}. However, such approaches implicitly assume the non-trivial conversion of text into triples.


C&S at different directions, using a common transmitted signal.

Joint MIMO-radar communications platform equipped with a reconfigurable transmit antenna array.

Joint system equipped with a phased array antenna implementing active radar sensing while communicating with a remote receiver.

Generate multibeams satisfying different beamforming requirements from C&S.

MIMO waveform orthogonality is exploited to permute the waveform across selected antenna grids and hence convey extra information bits.

Spatial modulation is applied to JCAS by allocating antenna elements based on the transmitted message, achieving increased communication rates by embedding additional data bits in the antenna selection.

Multibeam for Joint Communication and Sensing Using Steerable Analog Antenna Arrays

Abstract: Beamforming has great potential for joint communication and sensing (JCAS), which is becoming a demanding feature on many emerging platforms such as unmanned aerial vehicles and smart cars. Although beamforming has been extensively studied for communication and radar sensing respectively, its application in the joint system is not straightforward due to different beamforming requirements by communication and sensing. In this paper, we propose a novel multibeam framework using steerable analog antenna arrays, which allows seamless integration of communication and sensing. Different to conventional JCAS schemes that support JCAS using a single beam, our framework is based on the key innovation of multibeam technology: providing fixed subbeam for communication and packet-varying scanning subbeam for sensing, simultaneously from a single transmitting array. We provide a system architecture and protocols for the proposed framework, complying well with modern packet communication systems with multicarrier modulation. We also propose low-complexity and effective multibeam design and generation methods, which offer great flexibility in meeting different communication and sensing requirements. We further develop sensing parameter estimation algorithms using conventional digital Fourier transform and 1D compressive sensing techniques, matching well with the multibeam framework. Simulation results are provided and validate the effectiveness of our proposed framework, beamforming design methods and the sensing algorithms.

Dual-Function MIMO Radar Communications System Design Via Sparse Array Optimization

Abstract: Spectrum congestion and competition over frequency bandwidth could be alleviated by deploying dual-function radar-communications systems, where the radar platform presents itself as a system of opportunity to secondary communication functions. In this paper, we propose a new technique for communication information embedding into the emission of multiple-input multiple-output (MIMO) radar using sparse antenna array configurations. The phases induced by antenna displacements in a sensor array are unique, which makes array configuration feasible for symbol embedding. We also exploit the fact that in a MIMO radar system, the association of independent waveforms with the transmit antennas can change over different pulse repetition periods without impacting the radar functionality. We show that by reconfiguring sparse transmit array through antenna selection and reordering waveform-antenna paring, a data rate of megabits per second can be achieved for a moderate number of transmit antennas. To counteract practical implementation issues, we propose a regularized antenna selection based signaling scheme. The possible data rate is analyzed and the symbol/bit error rates are derived. Simulation examples are provided for performance evaluations and to demonstrate the effectiveness of proposed DFRC techniques.

Spatial Modulation for Joint Radar-Communications Systems: Design, Analysis, and Hardware Prototype

Abstract: Dual-function radar-communications (DFRC) systems implement radar and communication functionalities on a single platform. Jointly designing these subsystems can lead to substantial gains in performance as well as size, cost, and power consumption. In this paper, we propose a DFRC system, which utilizes generalized spatial modulation (GSM) to realize coexisting radar and communications waveforms. Our proposed GSM-based scheme, referred to as spatial modulation based communication-radar (SpaCoR) system, allocates antenna elements among the subsystems based on the transmitted message, thus achieving increased communication rates by embedding additional data bits in the antenna selection. We formulate the resulting signal models, and present a dedicated radar processing scheme. To evaluate the radar performance, we characterize the statistical properties of the transmit beam pattern. Then, we present a hardware prototype of the proposed DFRC system, demonstrating the feasibility of the scheme. Our results show that the proposed GSM system achieves improved communication performance compared to techniques utilizing fixed allocations operating at the same data rate. For the radar subsystem, our experiments show that the spatial agility induced by the GSM transmission improves the angular resolution and reduces the sidelobe level in the transmit beam pattern compared to using fixed antenna allocations.

Recent analytic results for antenna array design in JCAS.

A brief summary of recent analytic results of antenna array design in JCAS system is given in Table {{table:bcb391bf-46c2-48e5-b347-a9d0b14ab6e9}} .


Rethinking split manufacturing: An information-theoretic approach with secure layout techniques

Abstract: Split manufacturing is a promising technique to defend against fab-based malicious activities such as IP piracy, overbuilding, and insertion of hardware Trojans. However, a network flow-based proximity attack, proposed by Wang et al. (DAC'16) [1], has demonstrated that most prior art on split manufacturing is highly vulnerable. Here in this work, we present two practical layout techniques towards secure split manufacturing: (i) gate-level graph coloring and (ii) clustering of same-type gates. Our approach shows promising results against the advanced proximity attack, lowering its success rate by 5.27x, 3.19x, and 1.73x on average compared to the unprotected layouts when splitting at metal layers M1, M2, and M3, respectively. Also, it largely outperforms previous defense efforts; we observe on average 8x higher resilience when compared to representative prior art. At the same time, extensive simulations on ISCAS'85 and MCNC benchmarks reveal that our techniques incur an acceptable layout overhead. Apart from this empirical study, we provide — for the first time — a theoretical framework for quantifying the layout-level resilience against any proximity-induced information leakage. Towards this end, we leverage the notion of mutual information and provide extensive results to validate our model.

Concerted wire lifting: Enabling secure and cost-effective split manufacturing

Abstract: Here we advance the protection of split manufacturing (SM)-based layouts through the judicious and well-controlled handling of interconnects. Initially, we explore the cost-security trade-offs of SM, which are limiting its adoption. Aiming to resolve this issue, we propose effective and efficient strategies to lift nets to the BEOL. Towards this end, we design custom “elevating cells” which we also provide to the community. Further, we define and promote a new metric, Percentage of Netlist Recovery (PNR), which can quantify the resilience against gate-level theft of intellectual property (IP) in a manner more meaningful than established metrics. Our extensive experiments show that we outperform the recent protection schemes regarding security. For example, we reduce the correct connection rate to 0% for commonly considered benchmarks, which is a first in the literature. Besides, we induce reasonably low and controllable overheads on power, performance, and area (PPA). At the same time, we also help to lower the commercial cost incurred by SM.

Split Manufacturing Defenses.

Attacks toward Split Manufacturing showed promising results, as described in the previous section. A malicious attacker has the real potential to recover the missing BEOL connections. If the missing connections are successfully recovered, the security introduced by applying the technique is nullified. Therefore, straightforward Split Manufacturing is questioned by several works. Several authors propose defense techniques that augment the technique, i.e., techniques that, when used together with Split Manufacturing, do increase the achieved security against attacks. In Tab. {{table:242e86fd-ea9b-485d-8a96-fe695b0285c4}} , we compile a comprehensive list of defense techniques found in the literature. Each defense technique utilizes a different metric and Threat model, depending upon the type of attack they are trying to overcome. Since many of the studied defense techniques often introduce heavy PPA overheads, Tab. {{table:dcdcdf46-e5fb-4236-830d-454aeb20e6a8}}  also shows if the studied work assessed overheads and which ones were addressed.


Cross Modal Distillation for Supervision Transfer

Abstract: In this work we propose a technique that transfers supervision between images from different modalities. We use learned representations from a large labeled modality as a supervisory signal for training representations for a new unlabeled paired modality. Our method enables learning of rich representations for unlabeled modalities and can be used as a pre-training procedure for new modalities with limited labeled data. We show experimental results where we transfer supervision from labeled RGB images to unlabeled depth and optical flow images and demonstrate large improvements for both these cross modal supervision transfers. Code, data and pre-trained models are available at https://github.com/s-gupta/fast-rcnn/tree/distillation

Learning Deep Representations with Probabilistic Knowledge Transfer

Abstract: Knowledge Transfer (KT) techniques tackle the problem of transferring the knowledge from a large and complex neural network into a smaller and faster one. However, existing KT methods are tailored towards classification tasks and they cannot be used efficiently for other representation learning tasks. In this paper a novel knowledge transfer technique, that is capable of training a student model that maintains the same amount of mutual information between the learned representation and a set of (possible unknown) labels as the teacher model, is proposed. Apart from outperforming existing KT techniques, the proposed method allows for overcoming several limitations of existing methods providing new insight into KT as well as novel KT applications, ranging from knowledge transfer from handcrafted feature extractors to {cross-modal} KT from the textual modality into the representation extracted from the visual modality of the data.

Modality Distillation with Multiple Stream Networks for Action Recognition

Abstract: Diverse input data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while a (training) dataset could be accurately designed to include a variety of sensory inputs, it is often the case that not all modalities could be available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to learn robust representations leveraging multimodal data in the training stage, while considering limitations at test time, such as noisy or missing modalities. This paper presents a new approach for multimodal video action recognition, developed within the unified frameworks of distillation and privileged information, named generalized distillation. Particularly, we consider the case of learning representations from depth and RGB videos, while relying on RGB data only at test time. We propose a new approach to train an hallucination network that learns to distill depth features through multiplicative connections of spatiotemporal representations, leveraging soft labels and hard labels, as well as distance between feature maps. We report state-of-the-art results on video action classification on the largest multimodal dataset available for this task, the NTU RGB+D. Code available at https://github.com/ncgarcia/modality-distillation .

Cross-Modality Distillation: A case for Conditional Generative Adversarial Networks

Abstract: In this paper, we propose to use a Conditional Generative Adversarial Network (CGAN) for distilling (i.e. transferring) knowledge from sensor data and enhancing low-resolution target detection. In unconstrained surveillance settings, sensor measurements are often noisy, degraded, corrupted, and even missing/absent, thereby presenting a significant problem for multi-modal fusion. We therefore specifically tackle the problem of a missing modality in our attempt to propose an algorithm based on CGANs to generate representative information from the missing modalities when given some other available modalities. Despite modality gaps, we show that one can distill knowledge from one set of modalities to another. Moreover, we demonstrate that it achieves better performance than traditional approaches and recent teacher-student models.

Emotion Recognition in Speech using Cross-Modal Transfer in the Wild

Abstract: Obtaining large, human labelled speech datasets to train models for emotion recognition is a notoriously challenging task, hindered by annotation cost and label ambiguity. In this work, we consider the task of learning embeddings for speech classification without access to any form of labelled audio. We base our approach on a simple hypothesis: that the emotional content of speech correlates with the facial expression of the speaker. By exploiting this relationship, we show that annotations of expression can be transferred from the visual domain (faces) to the speech domain (voices) through cross-modal distillation. We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets. Code, models and data are available.

Cross-modal knowledge distillation for action recognition

Abstract: In this work, we address the problem how a network for action recognition that has been trained on a modality like RGB videos can be adapted to recognize actions for another modality like sequences of 3D human poses. To this end, we extract the knowledge of the trained teacher network for the source modality and transfer it to a small ensemble of student networks for the target modality. For the cross-modal knowledge distillation, we do not require any annotated data. Instead we use pairs of sequences of both modalities as supervision, which are straightforward to acquire. In contrast to previous works for knowledge distillation that use a KL-loss, we show that the cross-entropy loss together with mutual learning of a small ensemble of student networks performs better. In fact, the proposed approach for cross-modal knowledge distillation nearly achieves the accuracy of a student network trained with full supervision.

Compact Trilinear Interaction for Visual Question Answering

Abstract: In Visual Question Answering (VQA), answers have a great correlation with question meaning and visual contents. Thus, to selectively utilize image, question and answer information, we propose a novel trilinear interaction model which simultaneously learns high level associations between these three inputs. In addition, to overcome the interaction complexity, we introduce a multimodal tensor-based PARALIND decomposition which efficiently parameterizes trilinear teraction between the three inputs. Moreover, knowledge distillation is first time applied in Free-form Opened-ended VQA. It is not only for reducing the computational cost and required memory but also for transferring knowledge from trilinear interaction model to bilinear interaction model. The extensive experiments on benchmarking datasets TDIUC, VQA-2.0, and Visual7W show that the proposed compact trilinear interaction model achieves state-of-the-art results when using a single model on all three datasets.

Contrastive Representation Distillation

Abstract: Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation. Code: this http URL

A summary of cross-modal distillation with modalities, types of knowledge and distillation.

A summary of cross-modal distillation with different modalities, types of knowledge and distillation schemes is shown in Table {{table:34e5d438-97fa-4e5e-a1ff-56e99dbb252c}} . Specifically, it can be seen that knowledge distillation performs well in visual recognition tasks in the cross-modal scenarios. However, cross-modal knowledge transfer is a challenging study when there is a modality gap, e.g., lacking of the paired samples between different modalities.


A Large Self-Annotated Corpus for Sarcasm

Abstract: We introduce the Self-Annotated Reddit Corpus (SARC), a large corpus for sarcasm research and for training and evaluating systems for sarcasm detection. The corpus has 1.3 million sarcastic statements -- 10 times more than any previous dataset -- and many times more instances of non-sarcastic statements, allowing for learning in both balanced and unbalanced label regimes. Each statement is furthermore self-annotated -- sarcasm is labeled by the author, not an independent annotator -- and provided with user, topic, and conversation context. We evaluate the corpus for accuracy, construct benchmarks for sarcasm detection, and evaluate baseline methods.

Augmenting Data for Sarcasm Detection with Unlabeled Conversation Context

Abstract: We present a novel data augmentation technique, CRA (Contextual Response Augmentation), which utilizes conversational context to generate meaningful samples for training. We also mitigate the issues regarding unbalanced context lengths by changing the input-output format of the model such that it can deal with varying context lengths effectively. Specifically, our proposed model, trained with the proposed data augmentation technique, participated in the sarcasm detection task of FigLang2020, have won and achieves the best performance in both Reddit and Twitter datasets.

The standard datasets and the crawled dataset (for unlabeled augmentation) used in the experiments.

Table {{table:a738a2bc-33b4-49f8-81a0-6c36f50c9ec8}}  shows some samples generated from this technique. The quality of generated data depends undoubtedly on the degree of contextual conformity and similarity between the initial responses.
We find, however, that adding more data makes the quality of the augmented data better as the label transfer noise becomes attenuated. In summary, besides the standard datasets shown in Table {{table:702d57e0-3996-4985-bd44-5e252fe19ed6}} , we further crawled 100,000 texts from both Twitter and Reddit for the augmentation with unlabeled data.


Predicting Popularity of Online Videos Using Support Vector Regression

Abstract: In this work, we propose a regression method to predict the popularity of an online video measured by its number of views. Our method uses Support Vector Regression with Gaussian radial basis functions. We show that predicting popularity patterns with this approach provides more precise and more stable prediction results, mainly thanks to the nonlinear character of the proposed method as well as its robustness. We prove the superiority of our method against the state of the art using datasets containing almost 24 000 videos from YouTube and Facebook. We also show that using visual features, such as the outputs of deep neural networks or scene dynamics’ metrics, can be useful for popularity prediction before content publication. Furthermore, we show that popularity prediction accuracy can be improved by combining early distribution patterns with social and visual features and that social features represent a much stronger signal in terms of video popularity prediction than the visual ones.

Sequential Prediction of Social Media Popularity with Deep Temporal Context Networks

Abstract: Prediction of popularity has profound impact for social media, since it offers opportunities to reveal individual preference and public attention from evolutionary social systems. Previous research, although achieves promising results, neglects one distinctive characteristic of social data, i.e., sequentiality. For example, the popularity of online content is generated over time with sequential post streams of social media. To investigate the sequential prediction of popularity, we propose a novel prediction framework called Deep Temporal Context Networks (DTCN) by incorporating both temporal context and temporal attention into account. Our DTCN contains three main components, from embedding, learning to predicting. With a joint embedding network, we obtain a unified deep representation of multi-modal user-post data in a common embedding space. Then, based on the embedded data sequence over time, temporal context learning attempts to recurrently learn two adaptive temporal contexts for sequential popularity. Finally, a novel temporal attention is designed to predict new popularity (the popularity of a new user-post pair) with temporal coherence across multiple time-scales. Experiments on our released image dataset with about 600K Flickr photos demonstrate that DTCN outperforms state-of-the-art deep prediction algorithms, with an average of 21.51% relative performance improvement in the popularity prediction (Spearman Ranking Correlation).

Comparison of Popularity Prediction and Community Detection Techniques

Comparison: SOCIAL-AWARENESS, POPULARITY PREDICTION, AND COMMUNITY DETECTION

Table {{table:6d7bf6bb-c13b-48cd-8dfe-57c897c71c3e}}  lists the comparison of the state of the art literature on popularity prediction and community detection in edge networks.

Majority of the schemes listed in Table {{table:6d7bf6bb-c13b-48cd-8dfe-57c897c71c3e}}  are based on ML methods. ML methods are brought to service for either popularity prediction or community detection. It is noted that the edge caching techniques applied K-means clustering while the community detection techniques have applied DBSCAN in all listed works for optimal clustering of edge communities. Therefore, the application of density-based clustering in edge caching scenarios can be marked as future work for CHR optimization. The network scenario of online social networks in followed in most of the listed works. Varying parameters are applied for the evaluation of proposed works ranging from rank correlation {{cite:cbe31b3}} to graph density {{cite:bc8b2b3}}.


gilani-17 + cresci-rtbust + cresci-stock + kaiser-1 + kaiser-2

Online Human-Bot Interactions: Detection, Estimation, and Characterization

Abstract: Increasing evidence suggests that a growing amount of social media content is generated by autonomous entities known as social bots. In this work we present a framework to detect such entities on Twitter. We leverage more than a thousand features extracted from public data and meta-data about users: friends, tweet content and sentiment, network patterns, and activity time series. We benchmark the classification framework by using a publicly available dataset of Twitter bots. This training data is enriched by a manually annotated collection of active Twitter users that include both humans and bots of varying sophistication. Our models yield high accuracy and agreement with each other and can detect bots of different nature. Our estimates suggest that between 9% and 15% of active Twitter accounts are bots. Characterizing ties among accounts, we observe that simple bots tend to interact with bots that exhibit more human-like behaviors. Analysis of content flows reveals retweet and mention strategies adopted by bots to interact with different target groups. Using clustering analysis, we characterize several subclasses of accounts, including spammers, self promoters, and accounts that post content from connected applications.

The Paradigm-Shift of Social Spambots: Evidence, Theories, and Tools for the Arms Race

Abstract: Recent studies in social media spam and automation provide anecdotal argumentation of the rise of a new generation of spambots, so-called social spambots. Here, for the first time, we extensively study this novel phenomenon on Twitter and we provide quantitative evidence that a paradigm-shift exists in spambot design. First, we measure current Twitter's capabilities of detecting the new social spambots. Later, we assess the human performance in discriminating between genuine accounts, social spambots, and traditional spambots. Then, we benchmark several state-of-the-art techniques proposed by the academic literature. Results show that neither Twitter, nor humans, nor cutting-edge applications are currently capable of accurately detecting the new social spambots. Our results call for new approaches capable of turning the tide in the fight against this raising phenomenon. We conclude by reviewing the latest literature on spambots detection and we highlight an emerging common research trend based on the analysis of collective behaviors. Insights derived from both our extensive experimental campaign and survey shed light on the most promising directions of research and lay the foundations for the arms race against the novel social spambots. Finally, to foster research on this novel phenomenon, we make publicly available to the scientific community all the datasets used in this study.

Arming the public with artificial intelligence to counter social bots

Abstract: The increased relevance of social media in our daily life has been accompanied by efforts to manipulate online conversations and opinions. Deceptive social bots -- automated or semi-automated accounts designed to impersonate humans -- have been successfully exploited for these kinds of abuse. Researchers have responded by developing AI tools to arm the public in the fight against social bots. Here we review the literature on different types of bots, their impact, and detection methods. We use the case study of Botometer, a popular bot detection tool developed at Indiana University, to illustrate how people interact with AI countermeasures. A user experience survey suggests that bot detection has become an integral part of the social media experience for many users. However, barriers in interpreting the output of AI tools can lead to fundamental misunderstandings. The arms race between machine learning methods to develop sophisticated bots and effective countermeasures makes it necessary to update the training data and features of detection tools. We again use the Botometer case to illustrate both algorithmic and interpretability improvements of bot scores, designed to meet user expectations. We conclude by discussing how future AI developments may affect the fight between malicious bots and the public.

RTbust: Exploiting Temporal Patterns for Botnet Detection on Twitter

Abstract: Within OSNs, many of our supposedly online friends may instead be fake accounts called social bots, part of large groups that purposely re-share targeted content. Here, we study retweeting behaviors on Twitter, with the ultimate goal of detecting retweeting social bots.We collect a dataset of 10M retweets. We design a novel visualization that we leverage to highlight benign and malicious patterns of retweeting activity. In this way, we uncover a ?normal" retweeting pattern that is peculiar of human-operated accounts, and suspicious patterns related to bot activities. Then, we propose a bot detection technique that stems from the previous exploration of retweeting behaviors. Our technique, called Retweet-Buster (RTbust), leverages unsupervised feature extraction and clustering. An LSTM autoencoder converts the retweet time series into compact and informative latent feature vectors, which are then clustered with a hierarchical density-based algorithm. Accounts belonging to large clusters characterized by malicious retweeting patterns are labeled as bots. RTbust obtains excellent detection results, with F1=0.87, whereas competitors achieve F1?0.76.Finally, we apply RTbust to a large dataset of retweets, uncovering 2 previously unknown active botnets with hundreds of accounts.

Scalable and Generalizable Social Bot Detection through Data Selection

Abstract: Efficient and reliable social bot classification is crucial for detecting information manipulation on social media. Despite rapid development, state-of-the-art bot detection models still face generalization and scalability challenges, which greatly limit their applications. In this paper we propose a framework that uses minimal account metadata, enabling efficient analysis that scales up to handle the full stream of public tweets of Twitter in real time. To ensure model accuracy, we build a rich collection of labeled datasets for training and validation. We deploy a strict validation system so that model performance on unseen datasets is also optimized, in addition to traditional cross-validation. We find that strategically selecting a subset of training data yields better model accuracy and generalization than exhaustively training on all available data. Thanks to the simplicity of the proposed model, its logic can be interpreted to provide insights into social bot characteristics.

Detection of Novel Social Bots by Ensembles of Specialized Classifiers

Abstract: Malicious actors create inauthentic social media accounts controlled in part by algorithms, known as social bots, to disseminate misinformation and agitate online discussion. While researchers have developed sophisticated methods to detect abuse, novel bots with diverse behaviors evade detection. We show that different types of bots are characterized by different behavioral features. As a result, supervised learning techniques suffer severe performance deterioration when attempting to detect behaviors not observed in the training data. Moreover, tuning these models to recognize novel bots requires retraining with a significant amount of new annotations, which are expensive to obtain. To address these issues, we propose a new supervised learning method that trains classifiers specialized for each class of bots and combines their decisions through the maximum rule. The ensemble of specialized classifiers (ESC) can better generalize, leading to an average improvement of 56\% in F1 score for unseen accounts across datasets. Furthermore, novel bot behaviors are learned with fewer labeled examples during retraining. We deployed ESC in the newest version of Botometer, a popular tool to detect social bots in the wild, with a cross-validation AUC of 0.99.

Annotated datasets.

We considered various labeled datasets available through the Bot Repository (botometer.iuni.iu.edu/bot-repository).
Most of the datasets are annotated by humans, while others are created using automated techniques based on account behavior, filters on metadata, or more sophisticated procedures to achieve high precision (see Table {{table:cbf74730-2e58-48d2-9dd9-4a9fa4c7d4c6}}  for details).
For example, the vendor-purchased dataset contains fake followers purchased from several companies; accounts in cresci-stock show highly coordinated behavior that is extremely unlikely for human accounts; verified consists of randomly sampled verified accounts from the Twitter stream and is added to balance the datasets with only bots for our analysis. In addition to those datasets in the Bot Repository, we also collected accounts provided in a recent study conducted by Kaiser et al. {{cite:75043c8}}. These datasets made an assumption that all accounts belong to American and German politicians are being human accounts. They compliment this dataset with manually annotated German language bots and accounts listed in botwiki dataset.

Supervised bot detection methods achieve high accuracy based on in-domain cross-validation {{cite:91973e1}}.
To measure how recall deteriorates in cross-domain evaluation, we perform an experiment using four datasets selected from Table {{table:cbf74730-2e58-48d2-9dd9-4a9fa4c7d4c6}} : we train a model on one dataset and test it on another.
We use Random Forest classifiers with 100 decision trees. The fraction of trees outputting a positive label is calibrated using Platt's scaling {{cite:7b65ec8}} and binarized with a threshold of 0.5.
We use 5-fold cross-validation for in-domain classification; for consistency we split training and test samples in cross-domain cases as well, reporting average precision and recall.


Two-Stream Convolutional Networks for Action Recognition in Videos

Abstract:
We investigate architectures of discriminatively trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. We also aim to generalise the best performing hand-crafted features within a data-driven learning framework.

Our contribution is three-fold. First, we propose a two-stream ConvNet architecture which incorporates spatial and temporal networks. Second, we demonstrate that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, we show that multitask learning, applied to two different action classification datasets, can be used to increase the amount of training data and improve the performance on both. Our architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.

Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

Abstract: Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating $3\times3\times3$ convolutions with $1\times3\times3$ convolutional filters on spatial domain (equivalent to 2D CNN) plus $3\times1\times1$ convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networks. Our P3D ResNet achieves clear improvements on Sports-1M video classification dataset against 3D CNN and frame-based 2D CNN by 5.3% and 1.8%, respectively. We further examine the generalization performance of video representation produced by our pre-trained P3D ResNet on five different benchmarks and three different tasks, demonstrating superior performances over several state-of-the-art techniques.

Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection

Abstract: General human action recognition requires understanding of various visual cues. In this paper, we propose a network architecture that computes and integrates the most important visual cues for action recognition: pose, motion, and the raw images. For the integration, we introduce a Markov chain model which adds cues successively. The resulting approach is efficient and applicable to action classification as well as to spatial and temporal action localization. The two contributions clearly improve the performance over respective baselines. The overall approach achieves state-of-the-art action classification performance on HMDB51, J-HMDB and NTU RGB+D datasets. Moreover, it yields state-of-the-art spatio-temporal action localization results on UCF101 and J-HMDB.

The properties of action classification works

The Action Classification (AC) model plays such a role to map the spatiotemporal information to action categories. There are numerous AC studies considering the approaches of utilizing features and designing the model structure. In detail, Action Classification (AC) approaches could be divided into 5 categories, including RGB AC, RGB + Flow AC, Pose AC, RGB + Pose AC, and RGB + Flow + Pose AC, as shown in Figure {{figure:e3db3841-9004-4991-986a-040403ea52ec}} . Based on these 5 categories, we list the corresponding studies in Table {{table:a5087e9a-e893-44fb-a809-2731412048c3}} .


A Simple Framework for Contrastive Learning of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.

Self-Supervised Learning of Pretext-Invariant Representations

Abstract: The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations. Many pretext tasks lead to representations that are covariant with image transformations. We argue that, instead, semantic representations ought to be invariant under such transformations. Specifically, we develop Pretext-Invariant Representation Learning (PIRL, pronounced as `pearl') that learns invariant representations based on pretext tasks. We use PIRL with a commonly used pretext task that involves solving jigsaw puzzles. We find that PIRL substantially improves the semantic quality of the learned image representations. Our approach sets a new state-of-the-art in self-supervised learning from images on several popular benchmarks for self-supervised learning. Despite being unsupervised, PIRL outperforms supervised pre-training in learning image representations for object detection. Altogether, our results demonstrate the potential of self-supervised representations with good invariance properties.

Exploring Simple Siamese Representation Learning

Abstract: Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code will be made available.

Deep Clustering for Unsupervised Learning of Visual Features

Abstract: Clustering is a class of unsupervised learning methods that has been extensively applied and studied in computer vision. Little work has been done to adapt it to the end-to-end training of visual features on large scale datasets. In this work, we present DeepCluster, a clustering method that jointly learns the parameters of a neural network and the cluster assignments of the resulting features. DeepCluster iteratively groups the features with a standard clustering algorithm, k-means, and uses the subsequent assignments as supervision to update the weights of the network. We apply DeepCluster to the unsupervised training of convolutional neural networks on large datasets like ImageNet and YFCC100M. The resulting model outperforms the current state of the art by a significant margin on all the standard benchmarks.

Invariant Information Clustering for Unsupervised Image Classification and Segmentation

Abstract: We present a novel clustering objective that learns a neural network classifier from scratch, given only unlabelled data samples. The model discovers clusters that accurately match semantic classes, achieving state-of-the-art results in eight unsupervised clustering benchmarks spanning image classification and segmentation. These include STL10, an unsupervised variant of ImageNet, and CIFAR10, where we significantly beat the accuracy of our closest competitors by 6.6 and 9.5 absolute percentage points respectively. The method is not specialised to computer vision and operates on any paired dataset samples; in our experiments we use random transforms to obtain a pair from each image. The trained network directly outputs semantic labels, rather than high dimensional representations that need external processing to be usable for semantic clustering. The objective is simply to maximise mutual information between the class assignments of each pair. It is easy to implement and rigorously grounded in information theory, meaning we effortlessly avoid degenerate solutions that other clustering methods are susceptible to. In addition to the fully unsupervised mode, we also test two semi-supervised settings. The first achieves 88.8% accuracy on STL10 classification, setting a new global state-of-the-art over all existing methods (whether supervised, semi-supervised or unsupervised). The second shows robustness to 90% reductions in label coverage, of relevance to applications that wish to make use of small amounts of labels. github.com/xu-ji/IIC

A taxonomy of unsupervised deep learning methods, including three representative families in § – §.

Existing unsupervised deep learning models can be mainly grouped into three families: pretext tasks, discriminative models and generative models (Table {{table:46b32179-9950-4466-9938-637b08ab6c85}} ). Pretext tasks and discriminative models are also known as self-supervised learning, which drive model learning by a proxy protocol/task and construct pseudo label supervision to formulate unsupervised surrogate losses. Generative models is inherently unsupervised and explicitly models the data distribution to learn representations without label supervision. We review these models in §REF , §REF  and §REF .


Intensity-Free Learning of Temporal Point Processes

Abstract: Temporal point processes are the dominant paradigm for modeling sequences of events happening at irregular intervals. The standard way of learning in such models is by estimating the conditional intensity function. However, parameterizing the intensity function usually incurs several trade-offs. We show how to overcome the limitations of intensity-based approaches by directly modeling the conditional distribution of inter-event times. We draw on the literature on normalizing flows to design models that are flexible and efficient. We additionally propose a simple mixture model that matches the flexibility of flow-based models, but also permits sampling and computing moments in closed form. The proposed models achieve state-of-the-art performance in standard prediction tasks and are suitable for novel applications, such as learning sequence embeddings and imputing missing data.

Modeling the Intensity Function of Point Process Via Recurrent Neural Networks

Abstract: Event sequence, asynchronously generated with random timestamp, is ubiquitous among applications. The precise and arbitrary timestamp can carry important clues about the underlying dynamics, and has lent the event data fundamentally different from the time-series whereby series is indexed with fixed and equal time interval. One expressive mathematical tool for modeling event is point process. The intensity functions of many point processes involve two components: the background and the effect by the history. Due to its inherent spontaneousness, the background can be treated as a time series while the other need to handle the history events. In this paper, we model the background by a Recurrent Neural Network (RNN) with its units aligned with time series indexes while the history effect is modeled by another RNN whose units are aligned with asynchronous events to capture the long-range dynamics. The whole model with event type and timestamp prediction output layers can be trained end-to-end. Our approach takes an RNN perspective to point process, and models its background and history effect. For utility, our method allows a black-box treatment for modeling the intensity which is often a pre-defined parametric form in point processes. Meanwhile end-to-end training opens the venue for reusing existing rich techniques in deep network for point process modeling. We apply our model to the predictive maintenance problem using a log dataset by more than 1000 ATMs from a global bank headquartered in North America.

An Empirical Study: Extensive Deep Temporal Point Process

Abstract: Temporal point process as the stochastic process on continuous domain of time is commonly used to model the asynchronous event sequence featuring with occurrence timestamps. Thanks to the strong expressivity of deep neural networks, they are emerging as a promising choice for capturing the patterns in asynchronous sequences, in the context of temporal point process. In this paper, we first review recent research emphasis and difficulties in modeling asynchronous event sequences with deep temporal point process, which can be concluded into four fields: encoding of history sequence, formulation of conditional intensity function, relational discovery of events and learning approaches for optimization. We introduce most of recently proposed models by dismantling them into the four parts, and conduct experiments by remodularizing the first three parts with the same learning strategy for a fair empirical evaluation. Besides, we extend the history encoders and conditional intensity function family, and propose a Granger causality discovery framework for exploiting the relations among multi-types of events. Because the Granger causality can be represented by the Granger causality graph, discrete graph structure learning in the framework of Variational Inference is employed to reveal latent structures of the graph. Further experiments show that the proposed framework with latent graph discovery can both capture the relations and achieve an improved fitting and predicting performance.

Fully Neural Network based Model for General Temporal Point Processes

Abstract: A temporal point process is a mathematical model for a time series of discrete events, which covers various applications. Recently, recurrent neural network (RNN) based models have been developed for point processes and have been found effective. RNN based models usually assume a specific functional form for the time course of the intensity function of a point process (e.g., exponentially decreasing or increasing with the time since the most recent event). However, such an assumption can restrict the expressive power of the model. We herein propose a novel RNN based model in which the time course of the intensity function is represented in a general manner. In our approach, we first model the integral of the intensity function using a feedforward neural network and then obtain the intensity function as its derivative. This approach enables us to both obtain a flexible model of the intensity function and exactly evaluate the log-likelihood function, which contains the integral of the intensity function, without any numerical approximations. Our model achieves competitive or superior performances compared to the previous state-of-the-art methods for both synthetic and real datasets.

Transformer Hawkes Process

Abstract: Modern data acquisition routinely produce massive amounts of event sequence data in various domains, such as social media, healthcare, and financial markets. These data often exhibit complicated short-term and long-term temporal dependencies. However, most of the existing recurrent neural network based point process models fail to capture such dependencies, and yield unreliable prediction performance. To address this issue, we propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies and meanwhile enjoys computational efficiency. Numerical experiments on various datasets show that THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. Moreover, THP is quite general and can incorporate additional structural knowledge. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.

Wasserstein Learning of Deep Generative Point Process Models

Abstract: Point processes are becoming very popular in modeling asynchronous sequential data due to their sound mathematical foundation and strength in modeling a variety of real-world phenomena. Currently, they are often characterized via intensity function which limits model's expressiveness due to unrealistic assumptions on its parametric form used in practice. Furthermore, they are learned via maximum likelihood approach which is prone to failure in multi-modal distributions of sequences. In this paper, we propose an intensity-free approach for point processes modeling that transforms nuisance processes to a target one. Furthermore, we train the model using a likelihood-free leveraging Wasserstein distance between point processes. Experiments on various synthetic and real-world data substantiate the superiority of the proposed point process model over conventional ones.

Description of exisiting neural TPP methods, following {{cite:b339acf}}.

We review recently-proposed neural TPP models, and give a brief disciption of them in Table {{table:db38ea2b-5fbe-44fa-9223-369c2e5da1c9}} .
Most methods directly model the intensity function or probabilistic density function of the process,
while only WasGANTPP {{cite:9b0c0e3}} employed Wasserstein GAN as the probabilistic decoder, whose learning target is an approximation to the Wasserstein distance between the empirical and model distributions, and allows flexible sample generation.
The classical methods with closed-form likelihood usually show unsatisfactory performance, while SAHP, THP and WasGANTPP achieve improvements, as shown in Figure. REF .
SAHP and THP depends on numerical or Monte Carlo integration to approximate the likelihood because the probabilistic decoder has no closed form, leading to higher computational cost (shown in Figure REF ).
Besides, FNNInt, SAHP and THP do not allow flexible sampling, which limits the real-world application when one needs to draw out samples from the learned models.
Figure {{figure:fcc85cdd-f708-4154-9014-e1f7bfbb25b1}}  adn Table {{table:db38ea2b-5fbe-44fa-9223-369c2e5da1c9}}  give an intuitive demonstration of our motivation.

The methods of (1) {{formula:989a239f-a2db-4607-984c-f0bf561645ab}}  (4) have closed-form expectation. Mean of FNNInt, SAHP and THP is obtained by numerical integration, and mean of GNTPP is obtained by Monte Carlo integration thanks to its advantages in flexible sampling.
Note that it is possible to sample events from SAHP and THP using Ogata's thinning method {{cite:7cfd0d9}} since the intensity for both methods is monotonically decreasing between events. Samples can also be drawn from FNNInt model using numerical root-finding {{cite:4f60fdb}}, but these sampling methods designed especially for the three models have not yet been developed.
Therefore, flexible sampling is not allowed for FNNInt, SAHP and THP (Table {{table:db38ea2b-5fbe-44fa-9223-369c2e5da1c9}} ), so we do not report their `CRPS'.
In all the generative methods, the trick of log-noramlization is used: The input samples are firstly normalized by {{formula:b5da85d0-aac5-4b01-aff7-cca7e91c7b36}}  during training, and rescaled back by {{formula:eb40a656-e9af-43d6-9633-d2e2d5d56ef8}}  to make sure the sampled time intervals are positive.


Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Describing Textures in the Wild

Abstract: Patterns and textures are defining characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this analytical dimension in image understanding, we address the challenging problem of describing textures with semantic attributes. We identify a rich vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected in the wild.The resulting Describable Textures Dataset (DTD) is the basis to seek for the best texture representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and show that, surprisingly, it outperforms specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that the describable attributes are excellent texture descriptors, transferring between datasets and tasks; in particular, combined with IFV, they significantly outperform the state-of-the-art by more than 8 percent on both FMD and KTHTIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

The statistics of datasets.

In the problem of fine-grained image classification, we evaluate our method on the following widely used datasets: the CUB-200-2011 (CUB) {{cite:12c9128}}, the Stanford Cars (CAR) {{cite:175258d}}, the FGVC Aircraft (AIR) {{cite:fe8cffc}}, and the large-scale long-tailed iNaturalist 2018 {{cite:c337b49}}.
In our experiments, we follow the data splits provided in {{cite:3eeeab9}}, whose details are provided in Table {{table:3e6e3f68-d31d-4883-9b3d-5476046a9a6b}} , while the example images are visualized in Fig. {{figure:4729a998-9856-4e58-a127-a9b53e4b4d0f}} .

More experiments are conducted on the problem of generic image and texture classification, on two generic image classification benchmarks – the CIFAR {{cite:b01434a}} and the ImageNet {{cite:9548b2b}} and also two texture image datasets – the DTD {{cite:9a633d2}} and the MIT Scene (MIT) {{cite:9b101e0}}.
The dataset statistics are also summarized in Table {{table:3e6e3f68-d31d-4883-9b3d-5476046a9a6b}} , and training/testing splits for the DTD and the MIT datasets from their official websites are adopted in our experiments.
Only category labels of training data are adopted as supervision signals without using any additional prior information.

Robust Against Sparse and Imbalanced Data

In order to verify the robustness of our method against sparse and imbalanced data distributions, we conduct two more experiments on the CUB-200-2011 dataset with the ResNet50 backbone.
As Table {{table:3e6e3f68-d31d-4883-9b3d-5476046a9a6b}}  shown, the CUB-200-2011 dataset contains total 200 classes, where each class has a relative balanced distribution (about 30 training samples for each class), which thus requires to adapt the CUB dataset to a more challenging sparse and imbalanced setting.


TypeSQL: Knowledge-Based Type-Aware Neural Text-to-SQL Generation

Abstract: Interacting with relational databases through natural language helps users with any background easily query and analyze a vast amount of data. This requires a system that understands users’ questions and converts them to SQL queries automatically. In this paper, we present a novel approach TypeSQL which formats the problem as a slot filling task in a more reasonable way. In addition, TypeSQL utilizes type information to better understand rare entities and numbers in the questions. We experiment this idea on the WikiSQL dataset and outperform the prior art by 6% in much shorter time. We also show that accessing the content of databases can significantly improve the performance when users’ queries are not well-formed. TypeSQL can reach 82.6% accuracy, a 17.5% absolute improvement compared to the previous content-sensitive model.

Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing

Abstract: Research on parsing language to SQL has largely ignored the structure of the database (DB) schema, either because the DB was very simple, or because it was observed at both training and test time. In Spider, a recently-released text-to-SQL dataset, new and complex DBs are given at test time, and so the structure of the DB schema can inform the predicted SQL query. In this paper, we present an encoder-decoder semantic parser, where the structure of the DB schema is encoded with a graph neural network, and this representation is later used at both encoding and decoding time. Evaluation shows that encoding the schema structure improves our parser accuracy from 33.8% to 39.4%, dramatically above the current state of the art, which is at 19.7%.

RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers

Abstract: When translating natural language questions into SQL queries to answer questions from a database, contemporary semantic parsing models struggle to generalize to unseen database schemas. The generalization challenge lies in (a) encoding the database relations in an accessible way for the semantic parser, and (b) modeling alignment between database columns and their mentions in a given query. We present a unified framework, based on the relation-aware self-attention mechanism, to address schema encoding, schema linking, and feature representation within a text-to-SQL encoder. On the challenging Spider dataset this framework boosts the exact match accuracy to 57.2%, surpassing its best counterparts by 8.7% absolute improvement. Further augmented with BERT, it achieves the new state-of-the-art performance of 65.6% on the Spider leaderboard. In addition, we observe qualitative improvements in the model's understanding of schema linking and alignment. Our implementation will be open-sourced at https://github.com/Microsoft/rat-sql.

A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization

Abstract: We present SQLova, the first Natural-language-to-SQL (NL2SQL) model to achieve human performance in WikiSQL dataset. We revisit and discuss diverse popular methods in NL2SQL literature, take a full advantage of BERT {Devlin et al., 2018) through an effective table contextualization method, and coherently combine them, outperforming the previous state of the art by 8.2% and 2.5% in logical form and execution accuracy, respectively. We particularly note that BERT with a seq2seq decoder leads to a poor performance in the task, indicating the importance of a careful design when using such large pretrained models. We also provide a comprehensive analysis on the dataset and our model, which can be helpful for designing future NL2SQL datsets and models. We especially show that our model's performance is near the upper bound in WikiSQL, where we observe that a large portion of the evaluation errors are due to wrong annotations, and our model is already exceeding human performance by 1.3% in execution accuracy.

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Abstract: Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .

Typical methods used for encoding in text-to-SQL. The full table of existing methods and more details are listed in Table  in Appendix .

Various methods have been adopted to address the challenges of representing the meaning of questions, representing the structure for DB schema, and linking the DB content to question.
We group them into five categories, as shown in Table {{table:c9f941a8-594c-4ff3-a1ec-19605adae2b9}} .


A Blockchain Implementation Prototype for the Electronic Open Source Traceability of Wood along the Whole Supply Chain

Abstract: This is the first work to introduce the use of blockchain technology for the electronic traceability of wood from standing tree to final user. Infotracing integrates the information related to the product quality with those related to the traceability [physical and digital documents (Radio Frequency IDentification—RFID—architecture)] within an online information system whose steps (transactions) can be made safe to evidence of alteration through the blockchain. This is a decentralized and distributed ledger that keeps records of digital transactions in such a way that makes them accessible and visible to multiple participants in a network while keeping them secure without the need of a centralized certification organism. This work implements a blockchain architecture within the wood chain electronic traceability. The infotracing system is based on RFID sensors and open source technology. The entire forest wood supply chain was simulated from standing trees to the final product passing through tree cutting and sawmill process. Different kinds of Internet of Things (IoT) open source devices and tags were used, and a specific app aiming the forest operations was engineered to collect and store in a centralized database information (e.g., species, date, position, dendrometric and commercial information).

Using Software Product Lines to Create Blockchain Products: Application to Supply Chain Traceability

Abstract: In recent years, blockchain has been growing rapidly from a niche technology to a promising solution for many sectors, due to its unique properties that empower the design of innovative applications. Nevertheless, the development of blockchain applications is still a challenge. Due to the technological novelty, only a few developers are familiar with blockchain technologies and smart contracts. Others might face a steep learning curve or difficulties to reuse existing code to build blockchain applications. This study proposes a novel approach to tackle these issues, through software product line engineering. To support the approach, a web platform to configure and generate a blockchain application for on-chain traceability is introduced. First, a feature model has been designed to model core features of the chosen domain, based on the existing literature. Then, a configurator has been implemented to support the feature selection phase. Finally, a generator is able to ingest such configurations to generate on-the-shelf blockchain products. The generalizability of the contribution is validated by reproducing on-chain traceability applications proposed in the literature by using the platform. This work provides the first evidence that the implementation of blockchain applications using software product lines enhances the quality of produced applications and reduces the time to market.

Blockchain traceability research used to design and test the FM

The construction of a FM requires extensive knowledge of its associated domain.
In this study, this knowledge has been extracted from 5 different works that propose on-chain traceability solutions, called foundational set, shown in Table {{table:9e3c6107-8ada-4fcf-a628-0fa768783069}}  {{cite:1bb73e9}}, {{cite:4f52c93}}, {{cite:c9ca2ff}}, {{cite:d390bb9}}, {{cite:42f20c2}}.
From these papers, the features that were at least present twice (2-of-5) were included in the FM.
In some cases, they have been refined manually by adding subfeatures (e.g., adding CRUD methods to manage application participants).
This results in a FM, presented in the following subsections.
Note that this FM is not meant to be a complete representation of existing on-chain traceability features, but provides its most salient characteristics.
A complete analysis through a systematic literature review is left for future work.

The protocol used in the evaluation of the contribution is the following.
First, a sample of two studies has been chosen, called the Test Set in Table {{table:9e3c6107-8ada-4fcf-a628-0fa768783069}} .
This selection has been performed following two criteria: (1) the source code is available online and (2) the functional requirements of the application proposed in the study can be clearly identified and extracted.
Then, for each study, the following steps have been conducted: