Automatic Detection of Cyberbullying in Social Media Text

Abstract: While social media offer great communication opportunities, they also increase the vulnerability of young people to threatening situations online. Recent studies report that cyberbullying constitutes a growing problem among youngsters. Successful prevention depends on the adequate detection of potentially harmful messages and the information overload on the Web requires intelligent systems to identify potential risks automatically. The focus of this paper is on automatic cyberbullying detection in social media text by modelling posts written by bullies, victims, and bystanders of online bullying. We describe the collection and fine-grained annotation of a training corpus for English and Dutch and perform a series of binary classification experiments to determine the feasibility of automatic cyberbullying detection. We make use of linear support vector machines exploiting a rich feature set and investigate which information sources contribute the most for this particular task. Experiments on a holdout test set reveal promising results for the detection of cyberbullying-related posts. After optimisation of the hyperparameters, the classifier yields an F1-score of 64% and 61% for English and Dutch respectively, and considerably outperforms baseline systems based on keywords and word unigrams.

Machine learning and semantic analysis of in-game chat for cyberbullying

Abstract: One major problem with cyberbullying research is the lack of data, since researchers are traditionally forced to rely on survey data where victims and perpetrators self-report their impressions. In this paper, an automatic data collection system is presented that continuously collects in-game chat data from one of the most popular online multi-player games: World of Tanks. The data was collected and combined with other information about the players from available online data services. It presents a scoring scheme to enable identification of cyberbullying based on current research. Classification of the collected data was carried out using simple feature detection with SQL database queries and compared to classification from AI-based sentiment text analysis services that have recently become available and further against manually classified data using a custom-built classification client built for this paper. The simple SQL classification proved to be quite useful at identifying some features of toxic chat such as the use of bad language or racist sentiments, however the classification by the more sophisticated online sentiment analysis services proved to be disappointing. The results were then examined for insights into cyberbullying within this game and it was shown that it should be possible to reduce cyberbullying within the World of Tanks game by a significant factor by simply freezing the player's ability to communicate through the in-game chat function for a short period after the player is killed within a match. It was also shown that very new players are much less likely to engage in cyberbullying, suggesting that it may be a learned behaviour from other players.

A Comparative Analysis of Machine Learning Techniques for Cyberbullying Detection on Twitter

Abstract: The advent of social media, particularly Twitter, raises many issues due to a misunderstanding regarding the concept of freedom of speech. One of these issues is cyberbullying, which is a critical global issue that affects both individual victims and societies. Many attempts have been introduced in the literature to intervene in, prevent, or mitigate cyberbullying; however, because these attempts rely on the victims’ interactions, they are not practical. Therefore, detection of cyberbullying without the involvement of the victims is necessary. In this study, we attempted to explore this issue by compiling a global dataset of 37,373 unique tweets from Twitter. Moreover, seven machine learning classifiers were used, namely, Logistic Regression (LR), Light Gradient Boosting Machine (LGBM), Stochastic Gradient Descent (SGD), Random Forest (RF), AdaBoost (ADB), Naive Bayes (NB), and Support Vector Machine (SVM). Each of these algorithms was evaluated using accuracy, precision, recall, and F1 score as the performance metrics to determine the classifiers’ recognition rates applied to the global dataset. The experimental results show the superiority of LR, which achieved a median accuracy of around 90.57%. Among the classifiers, logistic regression achieved the best F1 score (0.928), SGD achieved the best precision (0.968), and SVM achieved the best recall (1.00).

Cyberbullying severity detection: A machine learning approach

Abstract: With widespread usage of online social networks and its popularity, social networking platforms have given us incalculable opportunities than ever before, and its benefits are undeniable. Despite benefits, people may be humiliated, insulted, bullied, and harassed by anonymous users, strangers, or peers. In this study, we have proposed a cyberbullying detection framework to generate features from Twitter content by leveraging a pointwise mutual information technique. Based on these features, we developed a supervised machine learning solution for cyberbullying detection and multi-class categorization of its severity in Twitter. In the study we applied Embedding, Sentiment, and Lexicon features along with PMI-semantic orientation. Extracted features were applied with Naïve Bayes, KNN, Decision Tree, Random Forest, and Support Vector Machine algorithms. Results from experiments with our proposed framework in a multi-class setting are promising both with respect to Kappa, classifier accuracy and f-measure metrics, as well as in a binary setting. These results indicate that our proposed framework provides a feasible solution to detect cyberbullying behavior and its severity in online social networks. Finally, we compared the results of proposed and baseline features with other machine learning algorithms. Findings of the comparison indicate the significance of the proposed features in cyberbullying detection.

Cyberbullying Detection in Social Networks Using Bi-GRU with Self-Attention Mechanism

Abstract: With the propagation of cyberbullying in social networks as a trending subject, cyberbullying detection has become a social problem that researchers are concerned about. Developing intelligent models and systems helps detect cyberbullying automatically. This work focuses on text-based cyberbullying detection because it is the commonly used information carrier in social networks and is the widely used feature in this regard studies. Motivated by the documented success of neural networks, we propose a complete model combining the bidirectional gated recurrent unit (Bi-GRU) and the self-attention mechanism. In detail, we introduce the design of a GRU cell and Bi-GRU’s advantage for learning the underlying relationships between words from both directions. Besides, we present the design of the self-attention mechanism and the benefit of this joining for achieving a greater performance of cyberbullying classification tasks. The proposed model could address the limitation of the vanishing and exploding gradient problems. We avoid using oversampling or downsampling on experimental data which could result in the overestimation of evaluation. We conduct a comparative assessment on two commonly used datasets, and the results show that our proposed method outperformed baselines in all evaluation metrics.

Summarization of cyberbullying detection study. Information that has not been provided by the authors is indicated by the symbol (-).

Fang, Yong et al. {{cite:08d0b24}}, conducted an experiment using three datasets of which two were collected from Twitter and the other from Wikipedia. The data was fed into a combination of a bidirectional GRU and a self-attention algorithm to detect textual cyberbullying content. It was observed in the study that the proposed methods performed tremendously high in all performance metrics. Table {{table:8c5f8c91-a521-4fc1-9f4d-d259667c1d67}}  is a representation of studies, metrics, approach categories, and classification algorithms used in cyberbullying detection.
The studies highlighted in the table are ordered based on the year of publication, 2017 to 2022, as mentioned in the inclusion criteria.


AI service placement in mobile edge intelligence

Multi-user collaborative DNN partitioning

Total time and energy consumption/Service placement decision, local CPU frequencies, uplink bandwidth, edge CPU frequency

Execution latency/Partitioning, computation resources

Fully utilize scarce wireless spectrum and edge computing resources in provisioning AIGC services

Providing insights for partitioning AIGC models under edge-mobile collaboration

Optimizing AI Service Placement and Resource Allocation in Mobile Edge Intelligence Systems

Abstract: Leveraging recent advances on mobile edge computing (MEC), edge intelligence has emerged as a promising paradigm to support mobile artificial intelligence (AI) applications at the network edge. In this paper, we consider the AI service placement problem in a multi-user MEC system, where the access point (AP) places the most up-to-date AI program at user devices to enable local computing/task execution at the user side. To fully utilize the stringent wireless spectrum and edge computing resources, the AP sends the AI service program to a user only when enabling local computing at the user yields a better system performance. We formulate a mixed-integer non-linear programming (MINLP) problem to minimize the total computation time and energy consumption of all users by jointly optimizing the service placement (i.e., which users to receive the program) and resource allocation (on local CPU frequencies, uplink bandwidth, and edge CPU frequency). To tackle the MINLP problem, we derive analytical expressions to calculate the optimal resource allocation decisions with low complexity. This allows us to efficiently obtain the optimal service placement solution by search-based algorithms such as meta-heuristic or greedy search algorithms. To enhance the algorithm scalability in large-sized networks, we further propose an ADMM (alternating direction method of multipliers) based method to decompose the optimization problem into parallel tractable MINLP subproblems. The ADMM method eliminates the need of searching in a high-dimensional space for service placement decisions and thus has a low computational complexity that grows linearly with the number of users. Simulation results show that the proposed algorithms perform extremely close to the optimum and significantly outperform the other representative benchmark algorithms.

Joint Multiuser DNN Partitioning and Computational Resource Allocation for Collaborative Edge Intelligence

Abstract: Mobile-edge computing (MEC) has emerged as a promising supporting architecture providing a variety of resources to the network edge, thus acting as an enabler for edge intelligence services empowering massive mobile and Internet-of-Things (IoT) devices with artificial intelligence (AI) capability. With the assistance of edge servers, user equipments (UEs) are able to run deep neural network (DNN)-based AI applications, which are generally resource hungry and computation intensive such that an individual UE can hardly afford by itself in real time. However, the resources in each individual edge server are typically limited. Therefore, any resource optimization involving edge servers is by nature a resource-constrained optimization problem and needs to be tackled in such a realistic context. Motivated by this observation, we investigate the optimization problem of DNN partitioning (an emerging DNN offloading scheme) in a realistic multiuser resource-constrained condition that rarely considered in previous works. Despite the extremely large solution space, we reveal several properties of this specific optimization problem of joint multi-UE DNN partitioning and computational resource allocation. We propose an algorithm called iterative alternating optimization (IAO) that can achieve the optimal solution in polynomial time. In addition, we present a rigorous theoretic analysis of our algorithm in terms of time complexity and performance under realistic estimation error. Moreover, we build a prototype that implements our framework and conducts extensive experiments using realistic DNN models, whose results demonstrate its effectiveness and efficiency.

Summary of scenarios, problems, benefits/challenges, and mathematical tools of edge resource allocation.

AIGC service provisioning based on edge intelligence is computationally and communication-intensive for resource-constrained edge servers and mobile devices {{cite:99ebf80}}. Specifically, AIGC users send service allocation requests to edge services. Upon receiving these AIGC requests, edge servers perform the AIGC tasks and deliver the output to users {{cite:cfa89e1}}. During this AIGC service provisioning interaction, model accuracy and resource consumption are the most common metrics. Consequently, significant efforts are being made to coordinate mobile devices and edge servers for deploying generative AI at mobile edge networks. As summarized in Table {{table:514d1941-2a4b-4e67-a354-04f957d5fdcc}} , several Key Performance Indicators (KPIs) for edge resource allocation in AIGC networks are presented below.


Solving General Arithmetic Word Problems

Abstract: This paper presents a novel approach to automatically solving arithmetic word problems. This is the first algorithmic approach that can handle arithmetic problems with multiple steps and operations, without depending on additional annotations or predefined templates. We develop a theory for expression trees that can be used to represent and evaluate the target arithmetic expressions; we use it to uniquely decompose the target arithmetic problem to multiple classification problems; we then compose an expression tree, combining these with world knowledge through a constrained inference framework. Our classifiers gain from the use of {\em quantity schemas} that supports better extraction of features. Experimental results show that our method outperforms existing systems, achieving state of the art performance on benchmark datasets of arithmetic word problems.

Training Verifiers to Solve Math Word Problems

Abstract: State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.

Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems

Abstract: Solving algebraic word problems requires executing a series of arithmetic operations---a program---to obtain a final answer. However, since programs can be arbitrarily complicated, inducing them directly from question-answer pairs is a formidable challenge. To make this task more feasible, we solve these problems by generating answer rationales, sequences of natural language and human-readable mathematical expressions that derive the final answer through a series of small steps. Although rationales do not explicitly specify programs, they provide a scaffolding for their structure via intermediate milestones. To evaluate our approach, we have created a new 100,000-sample dataset of questions, answers and rationales. Experimental results show that indirect supervision of program learning via answer rationales is a promising strategy for inducing arithmetic programs.

Are NLP Models really able to Solve Simple Math Word Problems?

Abstract: The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered “solved” with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Abstract: When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from ConceptNet (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

Abstract: A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of $\sim$66%.

Description of datasets used in our study.

We provide a summary of datasets used in our experiments in Table {{table:cd7c779f-3e5b-4006-ab4d-534acc92e1b2}} . We consider the 10 datasets from {{cite:24ea3a3}}, used to measure reasoning performance. For Last Letter Concatenation and Coin Flip, we use the publicly available data provided by {{cite:24ea3a3}}.


Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Abstract: We present Fashion-MNIST, a new dataset comprising of 28x28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set has 60,000 images and the test set has 10,000 images. Fashion-MNIST is intended to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms, as it shares the same image size, data format and the structure of training and testing splits. The dataset is freely available at this https URL

A Neural Representation of Sketch Drawings

Abstract: We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.

Results on various image classification benchmarks with required training time in GPU hours.

All images are normalized by subtracting the mean and dividing by the standard deviation.
For images with resolution higher than 64x64, they are scaled to 64x64 for NCEvolve.
For TAPAS, images are always scaled to 32x32.
The maximum GPU budget per dataset is divided into three categories: low, medium and high.
A dataset is assigned to one category based on the number of examples.
All datasets with at most 10K examples get a low GPU budget of 2 hours.
Datasets with at most 75K examples get the medium budget of 5 hours.
Finally, all other datasets get the high budget of at most 16 hours.
In contrast to the literature in the domain of automated architecture search, this budget contains both the search and training time.
State-of-the-art methods use double this budget only for the search followed by an expensive post-processing {{cite:869bef3}}.
We evaluate NeuNetS on 12 image classification benchmarks and report the results in Table {{table:50fd7d28-7dd7-4300-8029-f1a4b73551e2}} .


Peer-to-Peer Contact Tracing: Development of a Privacy-Preserving Smartphone App

Abstract: Background The novel coronavirus disease 2019 (COVID-19) pandemic is an urgent public health crisis, with epidemiologic models predicting severe consequences, including high death rates, if the virus is permitted to run its course without any intervention or response. Contact tracing using smartphone technology is a powerful tool that may be employed to limit disease transmission during an epidemic or pandemic; yet, contact tracing apps present significant privacy concerns regarding the collection of personal data such as location. Objective The aim of this study is to develop an effective contact tracing smartphone app that respects user privacy by not collecting location information or other personal data. Methods We propose the use of an anonymized graph of interpersonal interactions to conduct a novel form of contact tracing and have developed a proof-of-concept smartphone app that implements this approach. Additionally, we developed a computer simulation model that demonstrates the impact of our proposal on epidemic or pandemic outbreak trajectories across multiple rates of adoption. Results Our proof-of-concept smartphone app allows users to create “checkpoints” for contact tracing, check their risk level based on their past interactions, and anonymously self-report a positive status to their peer network. Our simulation results suggest that higher adoption rates of such an app may result in a better controlled epidemic or pandemic outbreak. Conclusions Our proposed smartphone-based contact tracing method presents a novel solution that preserves privacy while demonstrating the potential to suppress an epidemic or pandemic outbreak. This app could potentially be applied to the current COVID-19 pandemic as well as other epidemics or pandemics in the future to achieve a middle ground between drastic isolation measures and unmitigated disease spread.

A 5G NR based System Architecture for Real-Time Control with Batteryless RFID Sensors

Abstract: The fifth-generation wireless networking (5G) technologies have been developed to meet various time-critical use cases with ultra-reliable, low-latency and massive machine-type communications which are indispensable for tactile Internet applications. Recent advancements in very low-cost and batteryless radio-frequency identification (RFID) sensors have given promises of deploying a massive amount of such sensors for real-time sensing and control applications on a 5G New Radio (NR) network. However, the system design and performance of such applications have not been well studied. This paper proposes a novel system architecture for the representative batteryless RFID touch sensors in generic real-time control applications in a 5G NR mmWave environment. We will discuss the solution using edge computing nodes on the 5G NR base station to the implementation of the proposed system architecture. The real-time performance evaluation with the comparison of the Long-Term Evolution (LTE) networks has shown the effectiveness of the proposed system architecture.

Epidemic Contact Tracing via Communication Traces

Abstract: Traditional contact tracing relies on knowledge of the interpersonal network of physical interactions, where contagious outbreaks propagate. However, due to privacy constraints and noisy data assimilation, this network is generally difficult to reconstruct accurately. Communication traces obtained by mobile phones are known to be good proxies for the physical interaction network, and they may provide a valuable tool for contact tracing. Motivated by this assumption, we propose a model for contact tracing, where an infection is spreading in the physical interpersonal network, which can never be fully recovered; and contact tracing is occurring in a communication network which acts as a proxy for the first. We apply this dual model to a dataset covering 72 students over a 9 month period, for which both the physical interactions as well as the mobile communication traces are known. Our results suggest that a wide range of contact tracing strategies may significantly reduce the final size of the epidemic, by mainly affecting its peak of incidence. However, we find that for low overlap between the face-to-face and communication interaction network, contact tracing is only efficient at the beginning of the outbreak, due to rapidly increasing costs as the epidemic evolves. Overall, contact tracing via mobile phone communication traces may be a viable option to arrest contagious outbreaks.

Types of Sensing technologies for various transmission modes

Sensing Technologies for IoT-CTS Solutions

Detecting the exposure to infectious diseases through the person-to-person and object-to-person transmissions require sensing technologies. The possible sensing technologies are listed in Table {{table:aaaafe2b-bae2-4683-a374-b4aa7869099d}}  in alignment with the known methods of COVID-19 transmissions. The example sensor devices in Table {{table:aaaafe2b-bae2-4683-a374-b4aa7869099d}}  list the typical sensing devices, where “mobile” here generally refers to mobile devices with the presence of mobile networks, as it is often used to provide communication traces where possible social media traces from installed apps may also be utilized.

Generally speaking, these sensors shown in Table {{table:aaaafe2b-bae2-4683-a374-b4aa7869099d}}  can be operated in different modes, such as passive mode, proactive mode, and hybrid mode. The passive mode means sensors automatically collect tracing-related data. The proactive mode means sensors collect tracing data with users' full control. For example, using the QR code in Table {{table:aaaafe2b-bae2-4683-a374-b4aa7869099d}}  would fall in the proactive mode where active user participation is required. The hybrid mode means sensors can collect tracing data partially in the proactive mode and partially in the passive mode. In the hybrid mode, a user may configure which sensors can collect data without active interventions and which sensors can collect data under the user's control and permission. The sensing devices may take various form factors such as a wearable device. Although the privacy considerations can be addressed in these modes, the in-depth discussions are out of the scope of this paper.


Unpublished code provided by the authors

MesoNet: a Compact Facial Video Forgery Detection Network

Abstract: This paper presents a method to automatically and efficiently detect face tampering in videos, and particularly focuses on two recent techniques used to generate hyper-realistic forged videos: Deepfake and Face2Face. Traditional image forensics techniques are usually not well suited to videos due to the compression that strongly degrades the data. Thus, this paper follows a deep learning approach and presents two networks, both with a low number of layers to focus on the mesoscopic properties of images. We evaluate those fast networks on both an existing dataset and a dataset we have constituted from online videos. The tests demonstrate a very successful detection rate with more than 98% for Deepfake and 95% for Face2Face.

CNN-Generated Images Are Surprisingly Easy to Spot… for Now

Abstract: In this work we ask whether it is possible to create a "universal" detector for telling apart real images from these generated by a CNN, regardless of architecture or dataset used. To test this, we collect a dataset consisting of fake images generated by 11 different CNN-based image generator models, chosen to span the space of commonly used architectures today (ProGAN, StyleGAN, BigGAN, CycleGAN, StarGAN, GauGAN, DeepFakes, cascaded refinement networks, implicit maximum likelihood estimation, second-order attention super-resolution, seeing-in-the-dark). We demonstrate that, with careful pre- and post-processing and data augmentation, a standard image classifier trained on only one specific CNN generator (ProGAN) is able to generalize surprisingly well to unseen architectures, datasets, and training methods (including the just released StyleGAN2). Our findings suggest the intriguing possibility that today's CNN-generated images share some common systematic flaws, preventing them from achieving realistic image synthesis.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Abstract:
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.
To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.4% top-1 / 97.1% top-5 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at this https URL.

Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos

Abstract: Recent advances in media generation techniques have made it easier for attackers to create forged images and videos. State-of-the-art methods enable the real-time creation of a forged version of a single video obtained from a social network. Although numerous methods have been developed for detecting forged images and videos, they are generally targeted at certain domains and quickly become obsolete as new kinds of attacks appear. The method introduced in this paper uses a capsule network to detect various kinds of spoofs, from replay attacks using printed images or recorded videos to computer-generated videos using deep convolutional neural networks. It extends the application of capsule networks beyond their original intention to the solving of inverse graphics problems.

Exposing DeepFake Videos By Detecting Face Warping Artifacts

Abstract: In this work, we describe a new deep learning based method that can effectively distinguish AI-generated fake videos (referred to as {\em DeepFake} videos hereafter) from real videos. Our method is based on the observations that current DeepFake algorithm can only generate images of limited resolutions, which need to be further warped to match the original faces in the source video. Such transforms leave distinctive artifacts in the resulting DeepFake videos, and we show that they can be effectively captured by convolutional neural networks (CNNs). Compared to previous methods which use a large amount of real and DeepFake generated images to train CNN classifier, our method does not need DeepFake generated images as negative training examples since we target the artifacts in affine face warping as the distinctive feature to distinguish real and fake images. The advantages of our method are two-fold: (1) Such artifacts can be simulated directly using simple image processing operations on a image to make it as negative example. Since training a DeepFake model to generate negative examples is time-consuming and resource-demanding, our method saves a plenty of time and resources in training data collection; (2) Since such artifacts are general existed in DeepFake videos from different sources, our method is more robust compared to others. Our method is evaluated on two sets of DeepFake video datasets for its effectiveness in practice.

Face X-Ray for More General Face Forgery Detection

Abstract: In this paper we propose a novel image representation called face X-ray for detecting forgery in face images. The face X-ray of an input face image is a greyscale image that reveals whether the input image can be decomposed into the blending of two images from different sources. It does so by showing the blending boundary for a forged image and the absence of blending for a real image. We observe that most existing face manipulation methods share a common step: blending the altered face into an existing background image. For this reason, face X-ray provides an effective way for detecting forgery generated by most existing face manipulation algorithms. Face X-ray is general in the sense that it only assumes the existence of a blending step and does not rely on any knowledge of the artifacts associated with a specific face manipulation technique. Indeed, the algorithm for computing face X-ray can be trained without fake images generated by any of the state-of-the-art face manipulation methods. Extensive experiments show that face X-ray remains effective when applied to forgery generated by unseen face manipulation techniques, while most existing face forgery detection or deepfake detection algorithms experience a significant performance drop.

On the Detection of Digital Face Manipulation

Abstract: Detecting manipulated facial images and videos is an increasingly important topic in digital media forensics. As advanced face synthesis and manipulation methods are made available, new types of fake face representations are being created which have raised significant concerns for their use in social media. Hence, it is crucial to detect manipulated face images and localize manipulated regions. Instead of simply using multi-task learning to simultaneously detect manipulated images and predict the manipulated mask (regions), we propose to utilize an attention mechanism to process and improve the feature maps for the classification task. The learned attention maps highlight the informative regions to further improve the binary classification (genuine face v. fake face), and also visualize the manipulated regions. To enable our study of manipulated face detection and localization, we collect a large-scale database that contains numerous types of facial forgeries. With this dataset, we perform a thorough analysis of data-driven fake face detection. We show that the use of an attention mechanism improves facial forgery detection and manipulated region localization.

CORE: Consistent Representation Learning for Face Forgery Detection

Abstract: Face manipulation techniques develop rapidly and arouse widespread public concerns. Despite that vanilla convolutional neural networks achieve acceptable performance, they suffer from the overfitting issue. To relieve this issue, there is a trend to introduce some erasing-based augmentations. We find that these methods indeed attempt to implicitly induce more consistent representations for different augmentations via assigning the same label for different augmented images. However, due to the lack of explicit regularization, the consistency between different representations is less satisfactory. Therefore, we constrain the consistency of different representations explicitly and propose a simple yet effective framework, COnsistent REpresentation Learning (CORE). Specifically, we first capture the different representations with different augmentations, then regularize the cosine distance of the representations to enhance the consistency. Extensive experiments (in-dataset and cross-dataset) demonstrate that CORE performs favorably against state-of-the-art face forgery detection methods. Our code is available at https://github.com/niyunsheng/CORE.

UCF: Uncovering Common Features for Generalizable Deepfake Detection

Abstract: Deepfake detection remains a challenging task due to the difficulty of generalizing to new types of forgeries. This problem primarily stems from the overfitting of existing detection methods to forgery-irrelevant features and method-specific patterns. The latter has been rarely studied and not well addressed by previous works. This paper presents a novel approach to address the two types of overfitting issues by uncovering common forgery features. Specifically, we first propose a disentanglement framework that decomposes image information into three distinct components: forgery-irrelevant, method-specific forgery, and common forgery features. To ensure the decoupling of method-specific and common forgery features, a multi-task learning strategy is employed, including a multi-class classification that predicts the category of the forgery method and a binary classification that distinguishes the real from the fake. Additionally, a conditional decoder is designed to utilize forgery features as a condition along with forgery-irrelevant features to generate reconstructed images. Furthermore, a contrastive regularization technique is proposed to encourage the disentanglement of the common and specific forgery features. Ultimately, we only utilize the common forgery features for the purpose of generalizable deepfake detection. Extensive evaluations demonstrate that our framework can perform superior generalization than current state-of-the-art methods.

Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain

Abstract: The remarkable success in face forgery techniques has received considerable attention in computer vision due to security concerns. We observe that up-sampling is a necessary step of most face forgery techniques, and cumulative up-sampling will result in obvious changes in the frequency domain, especially in the phase spectrum. According to the property of natural images, the phase spectrum preserves abundant frequency components that provide extra information and complement the loss of the amplitude spectrum. To this end, we present a novel Spatial-Phase Shallow Learning (SPSL) method, which combines spatial image and phase spectrum to capture the up-sampling artifacts of face forgery to improve the transferability, for face forgery detection. And we also theoretically analyze the validity of utilizing the phase spectrum. Moreover, we notice that local texture information is more crucial than high-level semantic information for the face forgery detection task. So we reduce the receptive fields by shallowing the network to suppress high-level features and focus on the local region. Extensive experiments show that SPSL can achieve the state-of-the-art performance on cross-datasets evaluation as well as multi-class classification and obtain comparable results on single dataset evaluation.

Generalizing Face Forgery Detection with High-frequency Features

Abstract: Current face forgery detection methods achieve high accuracy under the within-database scenario where training and testing forgeries are synthesized by the same algorithm. However, few of them gain satisfying performance under the cross-database scenario where training and testing forgeries are synthesized by different algorithms. In this paper, we find that current CNN-based detectors tend to overfit to method-specific color textures and thus fail to generalize. Observing that image noises remove color textures and expose discrepancies between authentic and tampered regions, we propose to utilize the high-frequency noises for face forgery detection. We carefully devise three functional modules to take full advantage of the high-frequency features. The first is the multi-scale high-frequency feature extraction module that extracts high-frequency noises at multiple scales and composes a novel modality. The second is the residual-guided spatial attention module that guides the low-level RGB feature extractor to concentrate more on forgery traces from a new perspective. The last is the cross-modality attention module that leverages the correlation between the two complementary modalities to promote feature learning for each other. Comprehensive evaluations on several benchmark databases corroborate the superior generalization performance of our proposed method.

Summary of the compared deepfake detectors. For detectors without publicly available repositories, we undertake careful re-implementation, adhering to the instructions specified in the original papers.

Our benchmark has implemented a total of 15 established deepfake detection algorithms, as detailed in Tab. {{table:a3420fbe-d422-42dc-b48c-c8f32fcee7b1}} . The selection of these algorithms is guided by three criteria.
First, we prioritize methods that hold a classic status ({{formula:c4f63b77-42d5-40f7-9158-0f140af5a9b8}} , Xception), or those considered advanced, typically published in recent top-tier conferences or journals in computer vision or machine learning.
Second, our benchmark classifies detectors into three categories: naive detectors, spatial detectors, and frequency detectors. Our primary emphasis is on image forgery detection, hence, temporal-based detectors have not yet been incorporated. Moreover, we have refrained from including traditional detectors ({{formula:50d5d389-43f4-4070-94ea-11db7bdfd6e5}} , Headpose {{cite:2e4be85}}) due to their limited scalability to large-scale datasets, making them less suitable for our benchmark's objectives.
Third, we aim to include methods that are straightforward to implement and reproduce. We notice that several existing methods involve a series of steps, some of which are reliant on third-party algorithms or heuristic strategies. These methods usually have numerous hyper-parameters and are fraught with uncertainty, making their implementation and reproduction challenging. Therefore, these methods without open-source codes are intentionally excluded from our benchmark.
In summary, we provide a diverse set of detectors that are user-friendly and easily reproducible, such that the researchers' burden of extensive reimplementation could be alleviated.

The Training Module currently accommodates 15 detectors across three categories: naive detector, spatial detector, and frequency detector, all of which are shown in Tab. {{table:a3420fbe-d422-42dc-b48c-c8f32fcee7b1}} .
1) Naive Detector leverages various CNN architectures to directly detect forgeries without relying on additional manually designed features.
2) Spatial Detector builds upon the backbone of CNNs used in the Naive Detector and further explores manual-designed algorithms to detect deepfake.
3) Frequency Detector focuses on utilizing information from the frequency domain and extracting frequency artifacts for detection.
Each detector implemented in our benchmark is managed in a streamlined and efficient way, with a YAML config file created for each one. This allows users to easily set their desired parameters, {{formula:348ea25d-1423-4453-bab6-b45fc3c86555}} , batch size, learning rate, {{formula:9fa15cbd-d78e-4370-b855-2dafce75b3a6}} .
These detectors are trained on a unified trainer that records the metrics and losses during the training and evaluation process.
Thus, the training and evaluation processes, logging, and visualization are handled automatically, eliminating the need for manual specification.


Learning Review Representations from user and Product Level Information for Spam Detection

Abstract: Opinion spam has become a widespread problem in social media, where hired spammers write deceptive reviews to promote or demote products to mislead the consumers for profit or fame. Existing works mainly focus on manually designing discrete textual or behavior features, which cannot capture complex global semantics of reviews. Although recent works apply deep learning methods to learn review-level semantic features, their models ignore the impact of the user-level and product-level information on learning review semantics and the inherent user-review-product relationship information. In this paper, we propose a Hierarchical Fusion Attention Network (HFAN) to automatically learn the semantics of reviews from user and product level. Specifically, we design a multiattention unit to extract user(product)-related review information. Then, we use orthogonal decomposition and fusion attention to learn a user, review, and product representation from the review information. Finally, we take the review as a relation between user and product entity and apply TransH to jointly encode this relationship into review representation. Experimental results obtained more than 10% absolute precision improvement over the state-of-the-art performances on four real-world datasets, which show the effectiveness and versatility of the model.

Word Sense Disambiguation for 158 Languages using Word Embeddings Only

Abstract: Disambiguation of word senses in context is easy for humans, but is a major challenge for automatic approaches. Sophisticated supervised and knowledge-based models were developed to solve this task. However, (i) the inherent Zipfian distribution of supervised training instances for a given word and/or (ii) the quality of linguistic knowledge representations motivate the development of completely unsupervised and knowledge-free approaches to word sense disambiguation (WSD). They are particularly useful for under-resourced languages which do not have any resources for building either supervised and/or knowledge-based models. In this paper, we present a method that takes as input a standard pre-trained word embedding model and induces a fully-fledged word sense inventory, which can be used for disambiguation in context. We use this method to induce a collection of sense inventories for 158 languages on the basis of the original pre-trained fastText word embeddings by Grave et al., (2018), enabling WSD in these languages. Models and system are available online.

Sentiment Analysis Challenges in Education

Li et al. {{cite:989863b}} proposed a novel adaptive cross-contextual word embedding (ACWE) method to capture polysemous words in different contexts using topic modelling by defining a latent interpretable semantic space. A global word embedding was obtained based on an unsupervised cross-contextual probabilistic word embedding model and represents each word in the unified latent semantic space. An adaptive cross-contextual word embedding process is then defined to learn local word embedding for each polysemous word in different contexts. Jia et al. {{cite:d268e58}} proposed an effective topical word embedding (TWE)-based WSD method to generate topical word vectors for each word under each topic. The authors used LDA to extract high-quality topic models, and then generate contextual vectors generated for an ambiguous word to exploit the context. Based on the topic models and contextual vectors, topical word vectors were generated. Yenicelik et al. {{cite:e6803af}} used the BERT model to produce embedding vectors using quantitative analysis of linear separability and cluster organisation. The authors trained a simple linear classifier on top of the BERT embeddings to predict the semantic class and report accuracy. The challenges and trends in adopting sentiment analysis to education have been discussed and outlined in the research works in Tab. {{table:1318fc79-d4f0-403a-b5d8-d28712925d70}} .


Scalability in Perception for Autonomous Driving: Waymo Open Dataset

Abstract: The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the over-all viability of the technology. In an effort to help align the research community’s contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.

Argoverse: 3D Tracking and Forecasting With Rich Maps

Abstract: We present Argoverse, a dataset designed to support autonomous vehicle perception tasks including 3D tracking and motion forecasting. Argoverse includes sensor data collected by a fleet of autonomous vehicles in Pittsburgh and Miami as well as 3D tracking annotations, 300k extracted interesting vehicle trajectories, and rich semantic maps. The sensor data consists of 360 degree images from 7 cameras with overlapping fields of view, forward-facing stereo imagery, 3D point clouds from long range LiDAR, and 6-DOF pose. Our 290km of mapped lanes contain rich geometric and semantic metadata which are not currently available in any public dataset. All data is released under a Creative Commons license at Argoverse.org. In baseline experiments, we use map information such as lane direction, driveable area, and ground height to improve the accuracy of 3D object tracking. We use 3D object tracking to mine for more than 300k interesting vehicle trajectories to create a trajectory forecasting benchmark. Motion forecasting experiments ranging in complexity from classical methods (k-NN) to LSTMs demonstrate that using detailed vector maps with lane-level information substantially reduces prediction error. Our tracking and forecasting experiments represent only a superficial exploration of the potential of rich maps in robotic perception. We hope that Argoverse will enable the research community to explore these problems in greater depth.

One Thousand and One Hours: Self-driving Motion Prediction Dataset

Abstract: We present the largest self-driving dataset for motion prediction to date, with over 1,000 hours of data. This was collected by a fleet of 20 autonomous vehicles along a fixed route in Palo Alto, California over a four-month period. It consists of 170,000 scenes, where each scene is 25 seconds long and captures the perception output of the self-driving system, which encodes the precise positions and motions of nearby vehicles, cyclists, and pedestrians over time. On top of this, the dataset contains a high-definition semantic map with 15,242 labelled elements and a high-definition aerial view over the area. Together with the provided software kit, this collection forms the largest, most complete and detailed dataset to date for the development of self-driving, machine learning tasks such as motion forecasting, planning and simulation. The full dataset is available at this http URL.

Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks

Abstract: There has been significant research done on developing methods for improving robustness to distributional shift and uncertainty estimation. In contrast, only limited work has examined developing standard datasets and benchmarks for assessing these approaches. Additionally, most work on uncertainty estimation and robustness has developed new techniques based on small-scale regression or image classification tasks. However, many tasks of practical interest have different modalities, such as tabular data, audio, text, or sensor data, which offer significant challenges involving regression and discrete or continuous structured prediction. Thus, given the current state of the field, a standardized large-scale dataset of tasks across a range of modalities affected by distributional shifts is necessary. This will enable researchers to meaningfully evaluate the plethora of recently developed uncertainty quantification methods, as well as assessment criteria and state-of-the-art baselines. In this work, we propose the Shifts Dataset for evaluation of uncertainty estimates and robustness to distributional shift. The dataset, which has been collected from industrial sources and services, is composed of three tasks, with each corresponding to a particular data modality: tabular weather prediction, machine translation, and self-driving car (SDC) vehicle motion prediction. All of these data modalities and tasks are affected by real,"in-the-wild"distributional shifts and pose interesting challenges with respect to uncertainty estimation. In this work we provide a description of the dataset and baseline results for all tasks.

nuScenes: A Multimodal Dataset for Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.

Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting

Abstract: We introduce Argoverse 2 (AV2) - a collection of three datasets for perception and forecasting research in the self-driving domain. The annotated Sensor Dataset contains 1,000 sequences of multimodal data, encompassing high-resolution imagery from seven ring cameras, and two stereo cameras in addition to lidar point clouds, and 6-DOF map-aligned pose. Sequences contain 3D cuboid annotations for 26 object categories, all of which are sufficiently-sampled to support training and evaluation of 3D perception models. The Lidar Dataset contains 20,000 sequences of unlabeled lidar point clouds and map-aligned pose. This dataset is the largest ever collection of lidar sensor data and supports self-supervised learning and the emerging task of point cloud forecasting. Finally, the Motion Forecasting Dataset contains 250,000 scenarios mined for interesting and challenging interactions between the autonomous vehicle and other actors in each local scene. Models are tasked with the prediction of future motion for"scored actors"in each scenario and are provided with track histories that capture object location, heading, velocity, and category. In all three datasets, each scenario contains its own HD Map with 3D lane and crosswalk geometry - sourced from data captured in six distinct cities. We believe these datasets will support new and existing machine learning research problems in ways that existing datasets do not. All datasets are released under the CC BY-NC-SA 4.0 license.

Comparison of open AV datsets

Table.{{table:68cec976-1abd-4902-a7b7-41477763ba41}}  compares 6 popular open motion datasets. The recently-released Argoverse-2 {{cite:a118918}} motion forecasting dataset fulfils all these requirements. Argoverse-2 is collected by an automated fleet in 6 cities. The dataset contains 250k selected non-overlapping scenarios specifically focused on safety-critical long-tailed situations. The duration of each scenario is 11 with a uniform 10Hz sampling rate (0.1 time interval). Elements of high-definite maps, including vectorized maps and drivable areas, are also provided. For each road agent, timestamps, positions, velocities, and heading (yaw) directions are provided in the raw data. The developers also provide a Python toolkit "av2-api" https://github.com/argoverse/av2-api to read, process, and visualize the data.


Contact Patterns among High School Students

Abstract: Face-to-face contacts between individuals contribute to shape social networks and play an important role in determining how infectious diseases can spread within a population. It is thus important to obtain accurate and reliable descriptions of human contact patterns occurring in various day-to-day life contexts. Recent technological advances and the development of wearable sensors able to sense proximity patterns have made it possible to gather data giving access to time-varying contact networks of individuals in specific environments. Here we present and analyze two such data sets describing with high temporal resolution the contact patterns of students in a high school. We define contact matrices describing the contact patterns between students of different classes and show the importance of the class structure. We take advantage of the fact that the two data sets were collected in the same setting during several days in two successive years to perform a longitudinal analysis on two very different timescales. We show the high stability of the contact patterns across days and across years: the statistical distributions of numbers and durations of contacts are the same in different periods, and we observe a very high similarity of the contact matrices measured in different days or different years. The rate of change of the contacts of each individual from one day to the next is also similar in different years. We discuss the interest of the present analysis and data sets for various fields, including in social sciences in order to better understand and model human behavior and interactions in different contexts, and in epidemiology in order to inform models describing the spread of infectious diseases and design targeted containment strategies.

Estimating Potential Infection Transmission Routes in Hospital Wards Using Wearable Proximity Sensors

Abstract: Background Contacts between patients, patients and health care workers (HCWs) and among HCWs represent one of the important routes of transmission of hospital-acquired infections (HAI). A detailed description and quantification of contacts in hospitals provides key information for HAIs epidemiology and for the design and validation of control measures. Methods and Findings We used wearable sensors to detect close-range interactions (“contacts”) between individuals in the geriatric unit of a university hospital. Contact events were measured with a spatial resolution of about 1.5 meters and a temporal resolution of 20 seconds. The study included 46 HCWs and 29 patients and lasted for 4 days and 4 nights. 14,037 contacts were recorded overall, 94.1% of which during daytime. The number and duration of contacts varied between mornings, afternoons and nights, and contact matrices describing the mixing patterns between HCW and patients were built for each time period. Contact patterns were qualitatively similar from one day to the next. 38% of the contacts occurred between pairs of HCWs and 6 HCWs accounted for 42% of all the contacts including at least one patient, suggesting a population of individuals who could potentially act as super-spreaders. Conclusions Wearable sensors represent a novel tool for the measurement of contact patterns in hospitals. The collected data can provide information on important aspects that impact the spreading patterns of infectious diseases, such as the strong heterogeneity of contact numbers and durations across individuals, the variability in the number of contacts during a day, and the fraction of repeated contacts across days. This variability is however associated with a marked statistical stability of contact and mixing patterns across days. Our results highlight the need for such measurement efforts in order to correctly inform mathematical models of HAIs and use them to inform the design and evaluation of prevention strategies.

High-Resolution Measurements of Face-to-Face Contact Patterns in a Primary School

Abstract: Background Little quantitative information is available on the mixing patterns of children in school environments. Describing and understanding contacts between children at school would help quantify the transmission opportunities of respiratory infections and identify situations within schools where the risk of transmission is higher. We report on measurements carried out in a French school (6–12 years children), where we collected data on the time-resolved face-to-face proximity of children and teachers using a proximity-sensing infrastructure based on radio frequency identification devices. Methods and Findings Data on face-to-face interactions were collected on Thursday, October 1st and Friday, October 2nd 2009. We recorded 77,602 contact events between 242 individuals (232 children and 10 teachers). In this setting, each child has on average 323 contacts per day with 47 other children, leading to an average daily interaction time of 176 minutes. Most contacts are brief, but long contacts are also observed. Contacts occur mostly within each class, and each child spends on average three times more time in contact with classmates than with children of other classes. We describe the temporal evolution of the contact network and the trajectories followed by the children in the school, which constrain the contact patterns. We determine an exposure matrix aimed at informing mathematical models. This matrix exhibits a class and age structure which is very different from the homogeneous mixing hypothesis. Conclusions We report on important properties of the contact patterns between school children that are relevant for modeling the propagation of diseases and for evaluating control measures. We discuss public health implications related to the management of schools in case of epidemics and pandemics. Our results can help define a prioritization of control measures based on preventive measures, case isolation, classes and school closures, that could reduce the disruption to education during epidemics.

Narrative smoothing: Dynamic conversational network for the analysis of TV series plots

Abstract: Modern popular TV series often develop complex storylines spanning several seasons, but are usually watched in quite a discontinuous way. As a result, the viewer generally needs a comprehensive summary of the previous season plot before the new one starts. The generation of such summaries requires first to identify and characterize the dynamics of the series subplots. One way of doing so is to study the underlying social network of interactions between the characters involved in the narrative. The standard tools used in the Social Networks Analysis field to extract such a network rely on an integration of time, either over the whole considered period, or as a sequence of several time-slices. However, they turn out to be inappropriate in the case of TV series, due to the fact the scenes showed onscreen alternatively focus on parallel storylines, and do not necessarily respect a traditional chronology. In this article, we introduce narrative smoothing, a novel, still exploratory, network extraction method. It smooths the relationship dynamics based on the plot properties, aiming at solving some of the limitations present in the standard approaches. In order to assess our method, we apply it to a new corpus of 3 popular TV series, and compare it to both standard approaches. Our results are promising, showing narrative smoothing leads to more relevant observations when it comes to the characterization of the protagonists and their relationships. It could be used as a basis for further modeling the intertwined storylines constituting TV series plots.

Summary of real networks properties. n: number of nodes. m: number of different edges. e: number of interactions. t: number of timesteps. e/t: average number of observations per timestemps. e/m: average number of observation per edge. e/m/t: average probability to observe an existing edge at a given step.

In this section, we apply the same procedure on temporal networks corresponding to real datasets. We summarize information on those networks in table {{table:873830ac-4200-4293-995c-6774f58935cd}} . Three networks come from the SocioPatterns project {{cite:0c90082}}, SP-HS (High-School), SP-Hosp (Hospital) and SP-PS (Primary School). As already mentioned in section , they correspond to interactions collected between individuals every 20s. ENRON is a dataset of emails sent between employees. Temporal information is available at the level of the minute, over a period of about 3 years. Primates is a dataset of social interactions between primates, collected over 19 periods. GOT is a network of interactions between characters of a TV series (Game of Thrones) over several seasons, originally aggregated every 10 scenes.

All datasets are public, and available either through their original paper or though the network data repository {{cite:992b241}}, as reported in Table {{table:873830ac-4200-4293-995c-6774f58935cd}} .


Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Abstract: This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at https://github.com/microsoft/Swin-Transformer.

EVA-CLIP: Improved Training Techniques for CLIP at Scale

Abstract: Contrastive language-image pre-training, CLIP for short, has gained increasing attention for its potential in various scenarios. In this paper, we propose EVA-CLIP, a series of models that significantly improve the efficiency and effectiveness of CLIP training. Our approach incorporates new techniques for representation learning, optimization, and augmentation, enabling EVA-CLIP to achieve superior performance compared to previous CLIP models with the same number of parameters but significantly smaller training costs. Notably, our largest 5.0B-parameter EVA-02-CLIP-E/14+ with only 9 billion seen samples achieves 82.0 zero-shot top-1 accuracy on ImageNet-1K val. A smaller EVA-02-CLIP-L/14+ with only 430 million parameters and 6 billion seen samples achieves 80.4 zero-shot top-1 accuracy on ImageNet-1K val. To facilitate open access and open research, we release the complete suite of EVA-CLIP to the community at https://github.com/baaivision/EVA/tree/master/EVA-CLIP.

Model configurations of COSA. CC14M is a combination of CC3M {{cite:9622adc}}, CC12M {{cite:ac2957f}}, COCO {{cite:e02e392}}, SBU {{cite:806279d}}, and VG {{cite:a14d018}} datasets. The largest 1.2B COSA model is trained with a data bootstrapping strategy inspired by BLIP {{cite:5ae110b}}, and captions in training corpora (marked with `*') are generated by us through an additionally trained captioner, more details can be found in Appendix. The 400M image-text pairs used in CLIP and LAION-400M {{cite:c48be9c}} used in EVAClip are also included in the training examples statistics. LAION-102M is a randomly sampled subset from LAION-400M.

We train COSA models using the PyTorch framework and 64 Tesla V100 cards. To ensure a fair comparison with state-of-the-art video-language pretraining models and vision-language foundation models, we train four model variants with different parameter and data sizes, as illustrated in Table {{table:26a25fe2-0687-4474-9e64-149b8cdb6830}} . All models utilize BERT-Base as the text encoder. The initial learning rate is set to 1e-4, and a 10% warm-up strategy with a linear decay schedule is employed. The batch size is set to 2048. For ablation studies, we utilize the frozen CLIP/ViT-B/16 {{cite:0873556}} as the vision encoder for efficiency purposes, and the models are trained on CC3M for 30K steps with a batch size of 1024, unless specified otherwise.


Fish Recognition Based on Robust Features Extraction from Size and Shape Measurements Using Neural Network

Abstract: Problem statement: Image recognition is a challenging problem researchers had been research into this area for so long especially in the recent years, due to distortion, noise, segmentation errors, overlap and occlusion of objects in digital images. In our study, there are many fields concern with pattern recognition, for example, fingerprint verification, face recognition, iris discrimination, chromosome shape discrimination, optical character recognition, texture discrimination and speech recognition, the subject of pattern recognition appears. A system for recognizing isolated pattern of interest may be as an approach for dealing with such application. Scientists and engineers with interests in image processing and pattern recognition have developed various approaches to deal with digital image recognition problems such as, neural network, contour matching and statistics. Approach: In this study, our aim was to recognize an isolated pattern of interest in the image based on the combination between robust features extraction. Where depend on size and shape measurements, that were extracted by measuring the distance and geometrical measurements. Results: We presented a system prototype for dealing with such problem. The system started by acquiring an image containing pattern of fish, then the image features extraction is performed relying on size and shape measurements. Our system has been applied on 20 different fish families, each family has a different number of fish types and our sample consists of distinct 350 of fish images. These images were divided into two datasets: 257 training images and 93 testing images. An overall accuracy was obtained using the neural network associated with the back-propagation algorithm was 86% on the test dataset used. Conclusion: We developed a classifier for fish images recognition. We efficiently have chosen a features extraction method to fit our demands. Our classifier successfully design and implement a decision which performed efficiently without any problems. Eventually, the classifier is able to categorize the given fish into its cluster and categorize the clustered fish into its poison or non-poison fish and categorizes the poison and non-poison fish into its family.

A Feature Learning and Object Recognition Framework for Underwater Fish Images

Abstract: Live fish recognition is one of the most crucial elements of fisheries survey applications where the vast amount of data is rapidly acquired. Different from general scenarios, challenges to underwater image recognition are posted by poor image quality, uncontrolled objects and environment, and difficulty in acquiring representative samples. In addition, most existing feature extraction techniques are hindered from automation due to involving human supervision. Toward this end, we propose an underwater fish recognition framework that consists of a fully unsupervised feature learning technique and an error-resilient classifier. Object parts are initialized based on saliency and relaxation labeling to match object parts correctly. A non-rigid part model is then learned based on fitness, separation, and discrimination criteria. For the classifier, an unsupervised clustering approach generates a binary class hierarchy, where each node is a classifier. To exploit information from ambiguous images, the notion of partial classification is introduced to assign coarse labels by optimizing the benefit of indecision made by the classifier. Experiments show that the proposed framework achieves high accuracy on both public and self-collected underwater fish images with high uncertainty and class imbalance.

A list of computer vision studies for underwater fish classification between 2003-2021 using conventional classifiers and based on engineered features. The last column presents the work's achieved accuracy.

The Evolution of Fish Classification Algorithms over Two Decades

black
Computer vision for fish classification in the early 2000s and up to 2016, when first DL works started, has been mainly to manually extract fish features and then build classifiers that recognize these features. These conventional studies are listed, in a chronological order, in Table {{table:5c10574c-36eb-4d91-94fd-28b883c012f6}} .
Although there are many existing models, most of the classical non-DL models are based on local and engineered features. These include works using Haar features {{cite:e88042d}}, Scale-Invariant Feature Transform (SIFT) {{cite:b9bb162}}, and Histogram of Oriented Gradient (HOG) {{cite:61e76a3}}, which need hand-engineered algorithms. Because these algorithms are not suitable for recognizing images of untrained animals and cannot capture fish features from complex backgrounds, they usually use a large number of manually extracted samples to build classifiers.

black
As shown in Table {{table:5c10574c-36eb-4d91-94fd-28b883c012f6}} , support vector machines {{cite:7ec3aea}}, {{cite:e523690}}, {{cite:749d31b}}, {{cite:a732b3a}}, {{cite:f4c3774}}, {{cite:36445f4}}, {{cite:99f73a7}}, {{cite:f2f2673}}, {{cite:3a41839}} were one of the most commonly used classifiers for fish recognition, but they are prone to overfitting when trained with too many samples. This problem limits the scale of application. Another popular classification technique used in early works was backpropagation to train a simple feed-forward shallow neural network {{cite:8363bfb}}, {{cite:e2c630c}}, {{cite:d8120a1}}, {{cite:30071a2}}, {{cite:32f7a4d}}. Although this technique can handle simple samples, it is difficult to scale because of the neural network shallow layers, which will be explained in the next Section. Naive Bayes {{cite:3cf42fb}}, {{cite:1c8ed6b}}, {{cite:8d4620c}}, {{cite:68d1c76}} have also been used to classify fish since the early 2000s and up to 2017. The technique does not require much training data, and as shown in Table {{table:5c10574c-36eb-4d91-94fd-28b883c012f6}}  can reach good accuracy levels. Table {{table:5c10574c-36eb-4d91-94fd-28b883c012f6}}  also shows some other CV classification techniques, which while not as popular as the above-mentioned methods, could demonstrate good performance. However, it should be noted that, most of the CV techniques in Table {{table:5c10574c-36eb-4d91-94fd-28b883c012f6}} , were carefully engineered for their target datasets and are not capable of showing a similar performance level if used for another similar dataset. They will perhaps require an overhaul in their design, starting from manual feature engineering, to designing the detailed classification models.


Deep Reinforcement Learning-Feedforward NN

The fingerprinting database to unveil mapping between measured user location and RIS unit cells to maximize received signal strength

An optimization where the DDPG algorithm is responsible to search for the optimum action at each decision during the network learning phase

A novel usupervised learning architecture for beamforming design optimization

The architecture created an active learning scheme that quickly adapts to changing environmental conditions with high performance

The feed-forward NN fed with reflected symbols performing phase matrix optimization

Supervised Learning approach for the secrecy rate optimization at receiver and phase configuration optimization on the RIS side with reduced computational complexity

Deep reinforcement learning approach achieving higher SNR for noisy channels and mobile scenarios comparing to conventional techniques

A method that addresses the challenge in path loss optimization for RIS-aided terahertz communication having high molecular absorption and attenuation

The MLP-based NN addressing dynamic positioning of multiple RISs to overcome the storage and computational performance limitations

A NN trained by the approach of deep unsupervised learning optimizing both energy expenditure and phase configuration

Deep Reinforcement Learning-based novel architecture capable of learning channel behaviour

Indoor Signal Focusing with Deep Learning Designed Reconfigurable Intelligent Surfaces

Abstract: Reconfigurable Intelligent Surfaces (RISs) comprised of tunable unit elements have been recently considered in indoor communication environments for focusing signal reflections to intended user locations. However, the current proofs of concept require complex operations for the RIS configuration, which are mainly realized via wired control connections. In this paper, we present a deep learning method for efficient online wireless configuration of RISs when deployed in indoor communication environments. According to the proposed method, a database of coordinate fingerprints is implemented during an offline training phase. This fingerprinting database is used to train the weights and bias of a properly designed Deep Neural Network (DNN), whose role is to unveil the mapping between the measured coordinate information at a user location and the configuration of the RIS's unit cells that maximizes this user's received signal strength. During the online phase of the presented method, the trained DNN is fed with the measured position information at the target user to output the optimal phase configurations of the RIS for signal power focusing on this intended location. Our realistic simulation results using ray tracing on a three dimensional indoor environment demonstrate that the proposed DNN-based configuration method exhibits its merits for all considered cases, and effectively increases the achievable throughput at the target user location.

Optimization-driven Machine Learning for Intelligent Reflecting Surfaces Assisted Wireless Networks

Abstract: Intelligent reflecting surface (IRS) has been recently employed to reshape the wireless channels by controlling individual scattering elements' phase shifts, namely, passive beamforming. Due to the large size of scattering elements, the passive beamforming is typically challenged by the high computational complexity and inexact channel information. In this article, we focus on machine learning (ML) approaches for performance maximization in IRS-assisted wireless networks. In general, ML approaches provide enhanced flexibility and robustness against uncertain information and imprecise modeling. Practical challenges still remain mainly due to the demand for a large dataset in offline training and slow convergence in online learning. These observations motivate us to design a novel optimization-driven ML framework for IRS-assisted wireless networks, which takes both advantages of the efficiency in model-based optimization and the robustness in model-free ML approaches. By splitting the decision variables into two parts, one part is obtained by the outer-loop ML approach, while the other part is optimized efficiently by solving an approximate problem. Numerical results verify that the optimization-driven ML approach can improve both the convergence and the reward performance compared to conventional model-free learning approaches.

Resource Allocation In IRSs Aided MISO-NOMA Networks: A Machine Learning Approach

Abstract: A novel framework of intelligent reflecting surface (IRS)-aided multiple-input single-output (MISO) non-orthogonal multiple access (NOMA) network is proposed, where a base station (BS) serves multiple clusters with unfixed number of users in each cluster. The goal is to maximize the sum rate of all users by jointly optimizing the passive beamforming vector at the IRS, decoding order and power allocation coefficient vector, subject to the rate requirements of users. In order to tackle the formulated problem, a three-step approach is proposed. More particularly, a long short-term memory (LSTM) based algorithm is first adopted for predicting the mobility of users. Secondly, a K-means based Gaussian mixture model (K-GMM) algorithm is proposed for user clustering. Thirdly, a deep Q-network (DQN) based algorithm is invoked for jointly determining the phase shift matrix and power allocation policy. Simulation results are provided for demonstrating that the proposed algorithm outperforms the benchmarks, while the performance of IRS-NOMA system is better than IRS-OMA system.

Deep Reinforcement Learning for Intelligent Reflecting Surfaces: Towards Standalone Operation

Abstract: The promising coverage and spectral efficiency gains of intelligent reflecting surfaces (IRSs) are attracting increasing interest. In order to realize these surfaces in practice, however, several challenges need to be addressed. One of these main challenges is how to configure the reflecting coefficients on these passive surfaces without requiring massive channel estimation or beam training overhead. Earlier work suggested leveraging supervised learning tools to design the IRS reflection matrices. While this approach has the potential of reducing the beam training overhead, it requires collecting large datasets for training the neural network models. In this paper, we propose a novel deep reinforcement learning framework for predicting the IRS reflection matrices with minimal training overhead. Simulation results show that the proposed online learning framework can converge to the optimal rate that assumes perfect channel knowledge. This represents an important step towards realizing a standalone IRS operation, where the surface configures itself without any control from the infrastructure.

Deep Learning-based Phase Reconfiguration for Intelligent Reflecting Surfaces

Abstract: Intelligent reflecting surfaces (IRSs), consisting of reconfigurable metamaterials, have recently attracted attention as a promising cost-effective technology that can bring new features to wireless communications. These surfaces can be used to partially control the propagation environment and can potentially provide a power gain that is proportional to the square of the number of IRS elements when configured in a proper way. However, the configuration of the local phase matrix at the IRSs can be quite a challenging task since they are purposely designed to not have any active components, therefore, they are not able to process any pilot signal. In addition, a large number of elements at the IRS may create a huge training overhead. In this paper, we present a deep learning (DL) approach for phase reconfiguration at an IRS in order to learn and make use of the local propagation environment. The proposed method uses the received pilot signals reflected through the IRS to train the deep feedforward network. The performance of the proposed approach is evaluated and the numerical results are presented.

Truly Intelligent Reflecting Surface-Aided Secure Communication Using Deep Learning

Abstract: This paper considers machine learning for physical layer security design for communication in a challenging wireless environment. The radio environment is assumed to be programmable with the aid of a meta material-based intelligent reflecting surface (IRS) allowing customisable path loss, multi-path fading and interference effects. In particular, the fine-grained reflections from the IRS elements are exploited to create channel advantage for maximizing the secrecy rate at a legitimate receiver. A deep learning (DL) technique has been developed to tune the reflections of the IRS elements in real-time. Simulation results demonstrate that the DL approach yields comparable performance to the conventional approaches while significantly reducing the computational complexity.

On the Robustness of Deep Reinforcement Learning in IRS-Aided Wireless Communications Systems

Abstract: We consider an Intelligent Reflecting Surface (IRS)-aided multiple-input single-output (MISO) system for downlink transmission. We compare the performance of Deep Reinforcement Learning (DRL) and conventional optimization methods in finding optimal phase shifts of the IRS elements to maximize the user signal-to-noise (SNR) ratio. Furthermore, we evaluate the robustness of these methods to channel impairments and changes in the system. We demonstrate numerically that DRL solutions show more robustness to noisy channels and user mobility.

Hybrid Beamforming for RIS-Empowered Multi-hop Terahertz Communications: A DRL-based Method

Abstract: Wireless communication in the TeraHertz band (0.1--10 THz) is envisioned as one of the key enabling technologies for the future six generation (6G) wireless communication systems. However, very high propagation attenuations and molecular absorptions of THz frequencies often limit the signal transmission distance and coverage range. Benefited from the recent breakthrough on the reconfigurable intelligent surfaces (RIS) for realizing smart radio propagation environment, we propose a novel hybrid beamforming scheme for the multi-hop RIS-assisted communication networks to improve the coverage range at THz-band frequencies. We investigate the joint design of digital beamforming matrix at the BS and analog beamforming matrices at the RISs, by leveraging the recent advances in deep reinforcement learning (DRL) to combat the propagation loss. Simulation results show that our proposed scheme is able to improve 50\% more coverage range of THz communications compared with the benchmarks. Furthermore, it is also shown that our proposed DRL-based method is a state-of-the-art method to solve the NP-bard beamforming problem, especially when the signals at RIS-empowered THz communication networks experience multiple hops.

Phase Configuration Learning in Wireless Networks with Multiple Reconfigurable Intelligent Surfaces

Abstract: Reconfigurable Intelligent Surfaces (RISs) are recently gaining remarkable attention as a low-cost, hardware-efficient, and highly scalable technology capable of offering dynamic control of electro-magnetic wave propagation. Their envisioned dense deployment over various obstacles of the, otherwise passive, wireless communication environment has been considered as a revolutionary means to transform them into network entities with reconfigurable properties, providing increased environmental intelligence for diverse communication objectives. One of the major challenges with RIS-empowered wireless communications is the low-overhead dynamic configuration of multiple RISs, which according to the current hardware designs have very limited computing and storage capabilities. In this paper, we consider a typical communication pair between two nodes that is assisted by a plurality of RISs, and devise low-complexity supervised learning approaches for the RISs' phase configurations. By assuming common tunable phases in groups of each RIS's unit elements, we present multi-layer perceptron Neural Network (NN) architectures that can be trained either with positioning values or the instantaneous channel coefficients. We investigate centralized and individual training of the RISs, as well as their federation, and assess their computational requirements. Our simulation results, including comparisons with the optimal phase configuration scheme, showcase the benefits of adopting individual NNs at RISs for the link budget performance boosting.

Throughput maximization of an IRS-assisted wireless powered network with interference: A deep unsupervised learning approach

Abstract: We consider an intelligent reflecting surface (IRS)-assisted wireless powered communication network (WPCN) in which a multi antenna power beacon (PB) sends a dedicated energy signal to a wireless powered source. The source first harvests energy and then utilizing this harvested energy, it sends an information signal to destination where an external interference may also be present. For the considered system model, we formulated an analytical problem in which the objective is to maximize the throughput by jointly optimizing the energy harvesting (EH) time and IRS phase shift matrices. The optimization problem is high dimensional non-convex, thus a good quality solution can be obtained by invoking any state-of-the-art algorithm such as Genetic algorithm (GA). It is well-known that the performance of GA is generally remarkable, however it incurs a high computational complexity. To this end, we propose a deep unsupervised learning (DUL) based approach in which a neural network (NN) is trained very efficiently as time-consuming task of labeling a data set is not required. Numerical examples show that our proposed approach achieves a better performance-complexity trade-off as it is not only several times faster but also provides almost same or even higher throughput as compared to the GA.

Deep Learning-Empowered Predictive Beamforming for IRS-Assisted Multi-User Communications

Abstract: The realization of practical intelligent reflecting surface (IRS)-assisted multi-user communication (IRS-MUC) systems critically depends on the proper beamforming design exploiting accurate channel state information (CSI). However, channel estimation (CE) in IRS-MUC systems requires a significantly large training overhead due to the numerous reflection elements involved in IRS. In this paper, we adopt a deep learning approach to implicitly learn the historical channel features and directly predict the IRS phase shifts for the next time slot to maximize the average achievable sum-rate of an IRS-MUC system taking into account the user mobility. By doing this, only a low-dimension multiple-input single-output (MISO) CE is needed for transmit beamforming design, thus significantly reducing the CE overhead. To this end, a location-aware convolutional long short-term memory network (LA-CLNet) is first developed to facilitate predictive beamforming at IRS, where the convolutional and recurrent units are jointly adopted to exploit both the spatial and temporal features of channels simultaneously. Given the predictive IRS phase shift beamforming, an instantaneous CSI (ICSI)-aware fully-connected neural network (IA-FNN) is then proposed to optimize the transmit beamforming matrix at the access point. Simulation results demonstrate that the sum-rate performance achieved by the proposed method approaches that of the genie-aided scheme with the full perfect ICSI.

An overview of DL-based passive beamforming design studies.

Passive Beamforming Design with dl for System Optimization

Passive beamforming design is among the most critical problems for the RIS technology to be included in next generation of communication systems. The success of phase reconfiguration not only increases the quality of signal transformation but also increases the adaptability of RISs to different communication environments such as indoor/outdoor applications. Especially in highly dynamic channels, such as application scenarios with high mobility, the inadequacy of existing control systems has revealed the necessity of a new perspective. In preserving the fully passive nature of RISs with reduced costs and system complexity, truly intelligent approaches have a bright future for next-generation communications. Therefore, DL-based passive beamforming techniques have extensively been addressed by the literature as presented in Table {{table:e981473b-ac51-48b0-bd9b-19f0d3b12ed4}} .
Current DL-based studies on RIS-aided communication systems are composed of various NN structures depending on their advantages for target applications. mlp, a member of the feed-forward artificial-NN and one of the most basic types of DNN, can be given as an instance. {{cite:6069d7a}} offers an mlp architecture for beamforming design on the RIS side. The considered RIS scheme, which has a hybrid structure in which active and passive elements are present simultaneously, performs pilot training with randomly distributed active components. The trained data set, in which the beamforming on the RIS side is established as an input-output pair, is built with the supervised learning scheme. The use of active elements can be considered as a drawback of this study. The possible advantages of the supervised learning-based MLP algorithm with fully passive RIS have been inspected in {{cite:5b56adc}} to overcome the storage and computational performance limitations in the dynamic positioning of multiple RISs. The MLP-based NN has fed by dynamic channel information as well as RIS positioning information. The proposed position-trained NN results are superior to benchmark results retrieved by exhaustive study.