How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection

Abstract: The introduction of ChatGPT has garnered widespread attention in both academic and industrial communities. ChatGPT is able to respond effectively to a wide range of human questions, providing fluent and comprehensive answers that significantly surpass previous public chatbots in terms of security and usefulness. On one hand, people are curious about how ChatGPT is able to achieve such strength and how far it is from human experts. On the other hand, people are starting to worry about the potential negative impacts that large language models (LLMs) like ChatGPT could have on society, such as fake news, plagiarism, and social security issues. In this work, we collected tens of thousands of comparison responses from both human experts and ChatGPT, with questions ranging from open-domain, financial, medical, legal, and psychological areas. We call the collected dataset the Human ChatGPT Comparison Corpus (HC3). Based on the HC3 dataset, we study the characteristics of ChatGPT's responses, the differences and gaps from human experts, and future directions for LLMs. We conducted comprehensive human evaluations and linguistic analyses of ChatGPT-generated content compared with that of humans, where many interesting results are revealed. After that, we conduct extensive experiments on how to effectively detect whether a certain text is generated by ChatGPT or humans. We build three different detection systems, explore several key factors that influence their effectiveness, and evaluate them in different scenarios. The dataset, code, and models are all publicly available at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection.

Defending Against Neural Fake News

Abstract: Recent progress in natural language generation has raised dual-use concerns. While applications like summarization and translation are positive, the underlying technology also might enable adversaries to generate neural fake news: targeted propaganda that closely mimics the style of real news. Modern computer security relies on careful threat modeling: identifying potential threats and vulnerabilities from an adversary's point of view, and exploring potential mitigations to these threats. Likewise, developing robust defenses against neural fake news requires us first to carefully investigate and characterize the risks of these models. We thus present a model for controllable text generation called Grover. Given a headline like `Link Found Between Vaccines and Autism,' Grover can generate the rest of the article; humans find these generations to be more trustworthy than human-written disinformation. Developing robust verification techniques against generators like Grover is critical. We find that best current discriminators can classify neural fake news from real, human-written, news with 73% accuracy, assuming access to a moderate level of training data. Counterintuitively, the best defense against Grover turns out to be Grover itself, with 92% accuracy, demonstrating the importance of public release of strong generators. We investigate these results further, showing that exposure bias -- and sampling strategies that alleviate its effects -- both leave artifacts that similar discriminators can pick up on. We conclude by discussing ethical issues regarding the technology, and plan to release Grover publicly, helping pave the way for better detection of neural fake news.

TweepFake: About detecting deepfake tweets

Abstract: The recent advances in language modeling significantly improved the generative capabilities of deep neural models: in 2019 OpenAI released GPT-2, a pre-trained language model that can autonomously generate coherent, non-trivial and human-like text samples. Since then, ever more powerful text generative models have been developed. Adversaries can exploit these tremendous generative capabilities to enhance social bots that will have the ability to write plausible deepfake messages, hoping to contaminate public debate. To prevent this, it is crucial to develop deepfake social media messages detection systems. However, to the best of our knowledge no one has ever addressed the detection of machine-generated texts on social networks like Twitter or Facebook. With the aim of helping the research in this detection field, we collected the first dataset of real deepfake tweets, TweepFake. It is real in the sense that each deepfake tweet was actually posted on Twitter. We collected tweets from a total of 23 bots, imitating 17 human accounts. The bots are based on various generation techniques, i.e., Markov Chains, RNN, RNN+Markov, LSTM, GPT-2. We also randomly selected tweets from the humans imitated by the bots to have an overall balanced dataset of 25,572 tweets (half human and half bots generated). The dataset is publicly available on Kaggle. Lastly, we evaluated 13 deepfake text detection methods (based on various state-of-the-art approaches) to both demonstrate the challenges that Tweepfake poses and create a solid baseline of detection techniques. We hope that TweepFake can offer the opportunity to tackle the deepfake detection on social media messages as well.

OpenAI Gym

Abstract: OpenAI Gym is a toolkit for reinforcement learning research. It includes a growing collection of benchmark problems that expose a common interface, and a website where people can share their results and compare the performance of algorithms. This whitepaper discusses the components of OpenAI Gym and the design decisions that went into the software.

TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation

Abstract: Recent progress in generative language models has enabled machines to generate astonishingly realistic texts. While there are many legitimate applications of such models, there is also a rising need to distinguish machine-generated texts from human-written ones (e.g., fake news detection). However, to our best knowledge, there is currently no benchmark environment with datasets and tasks to systematically study the so-called "Turing Test" problem for neural text generation methods. In this work, we present the TuringBench benchmark environment, which is comprised of (1) a dataset with 200K human- or machine-generated samples across 20 labels {Human, GPT-1, GPT-2_small, GPT-2_medium, GPT-2_large, GPT-2_xl, GPT-2_PyTorch, GPT-3, GROVER_base, GROVER_large, GROVER_mega, CTRL, XLM, XLNET_base, XLNET_large, FAIR_wmt19, FAIR_wmt20, TRANSFORMER_XL, PPLM_distil, PPLM_gpt2}, (2) two benchmark tasks -- i.e., Turing Test (TT) and Authorship Attribution (AA), and (3) a website with leaderboards. Our preliminary experimental results using TuringBench show that FAIR_wmt20 and GPT-3 are the current winners, among all language models tested, in generating the most human-like indistinguishable texts with the lowest F1 score by five state-of-the-art TT detection models. The TuringBench is available at: https://turingbench.ist.psu.edu/

Representative Benchmarking Datasets

This section introduces several noteworthy benchmarking datasets for detecting LLM-generated texts. One prominent example is the work by Guo et al. {{cite:15b3a15}}, where the authors constructed the dataset Human ChatGPT Comparison Corpus (HC3), specifically designed to distinguish text generated by ChatGPT in question-answering tasks in both English and Chinese languages. The dataset comprises 37,175 questions across diverse domains, such as open-domain, computer science, finance, medicine, law, and psychology. These questions and corresponding human answers were sourced from publicly available question-answering datasets and wiki text. The researchers obtained responses generated by ChatGPT by presenting the same questions to the model and collecting its outputs. The evaluation results demonstrated that a RoBERTa-based detector achieved the highest performance, with F1 scores of 99.79% for paragraph-level detection and 98.43% for sentence-level detection of English text. Table {{table:4513ac41-bad3-4f91-a40a-255690f87377}}  presents more representative public benchmarking datasets for detecting different LLMs across various domains.


Linear Regression, Support Vector Machine

Length, Semantic, Glove, Elmo, InferSent, Phonetic, Frequency, POS

Frequency, Psycholinguistic, Sentence, Bigram Association

Length, Frequency, Character N-Grams, Psycholinguistic, POS

Transformers, Word Embeddings, Character Wmbeddings, Length, Psycholinguistic

BigGreen at SemEval-2021 Task 1: Lexical Complexity Prediction with Assembly Models

Abstract: This paper describes a system submitted by team BigGreen to LCP 2021 for predicting the lexical complexity of English words in a given context. We assemble a feature engineering-based model with a deep neural network model founded on BERT. While BERT itself performs competitively, our feature engineering-based model helps in extreme cases, eg. separating instances of easy and neutral difficulty. Our handcrafted features comprise a breadth of lexical, semantic, syntactic, and novel phonological measures. Visualizations of BERT attention maps offer insight into potential features that Transformers models may learn when fine-tuned for lexical complexity prediction. Our ensembled predictions score reasonably well for the single word subtask, and we demonstrate how they can be harnessed to perform well on the multi word expression subtask too.

IITK@LCP at SemEval 2021 Task 1: Classification for Lexical Complexity Regression Task

Abstract: This paper describes our contribution to SemEval 2021 Task 1: Lexical Complexity Prediction. In our approach, we leverage the ELECTRA model and attempt to mirror the data annotation scheme. Although the task is a regression task, we show that we can treat it as an aggregation of several classification and regression models. This somewhat counter-intuitive approach achieved an MAE score of 0.0654 for Sub-Task 1 and MAE of 0.0811 on Sub-Task 2. Additionally, we used the concept of weak supervision signals from Gloss-BERT in our work, and it significantly improved the MAE score in Sub-Task 1.

LAST at SemEval-2021 Task 1: Improving Multi-Word Complexity Prediction Using Bigram Association Measures

Abstract: This paper describes the system developed by the Laboratoire d'analyse statistique des textes (LAST) for the Lexical Complexity Prediction shared task at SemEval-2021. The proposed system is made up of a LightGBM model fed with features obtained from many word frequency lists, published lexical norms and psychometric data. For tackling the specificity of the multi-word task, it uses bigram association measures. Despite that the only contextual feature used was sentence length, the system achieved an honorable performance in the multi-word task, but poorer in the single word task. The bigram association measures were found useful, but to a limited extent.

LCP-RIT at SemEval-2021 Task 1: Exploring Linguistic Features for Lexical Complexity Prediction

Abstract: This paper describes team LCP-RIT's submission to the SemEval-2021 Task 1: Lexical Complexity Prediction (LCP). The task organizers provided participants with an augmented version of CompLex (Shardlow et al., 2020), an English multi-domain dataset in which words in context were annotated with respect to their complexity using a five point Likert scale. Our system uses logistic regression and a wide range of linguistic features (e.g. psycholinguistic features, n-grams, word frequency, POS tags) to predict the complexity of single words in this dataset. We analyze the impact of different linguistic features in the classification performance and we evaluate the results in terms of mean absolute error, mean squared error, Pearson correlation, and Spearman correlation.

OCHADAI-KYOTO at SemEval-2021 Task 1: Enhancing Model Generalization and Robustness for Lexical Complexity Prediction

Abstract: We propose an ensemble model for predicting the lexical complexity of words and multiword expressions (MWEs). The model receives as input a sentence with a target word or MWEand outputs its complexity score. Given that a key challenge with this task is the limited size of annotated data, our model relies on pretrained contextual representations from different state-of-the-art transformer-based language models (i.e., BERT and RoBERTa), and on a variety of training methods for further enhancing model generalization and robustness:multi-step fine-tuning and multi-task learning, and adversarial training. Additionally, we propose to enrich contextual representations by adding hand-crafted features during training. Our model achieved competitive results and ranked among the top-10 systems in both sub-tasks.

UPB at SemEval-2021 Task 1: Combining Deep Learning and Hand-Crafted Features for Lexical Complexity Prediction

Abstract: Reading is a complex process which requires proper understanding of texts in order to create coherent mental representations. However, comprehension problems may arise due to hard-to-understand sections, which can prove troublesome for readers, while accounting for their specific language skills. As such, steps towards simplifying these sections can be performed, by accurately identifying and evaluating difficult structures. In this paper, we describe our approach for the SemEval-2021 Task 1: Lexical Complexity Prediction competition that consists of a mixture of advanced NLP techniques, namely Transformer-based language models, pre-trained word embeddings, Graph Convolutional Networks, Capsule Networks, as well as a series of hand-crafted textual complexity features. Our models are applicable on both subtasks and achieve good performance results, with a MAE below 0.07 and a Person correlation of .73 for single word identification, as well as a MAE below 0.08 and a Person correlation of .79 for multiple word targets. Our results are just 5.46% and 6.5% lower than the top scores obtained in the competition on the first and the second subtasks, respectively.

Systems submitted to LCP–2021 in alphabetical order as shown in {{cite:306a4cd}}.

LCP has been the focus of several international competitions, known as shared–tasks. These shared–tasks have been described throughout the following sections (Sections REF  to REF ). Further detail regarding the architecture, development, and evolution of the systems submitted to these shared–tasks has been provided in Section . In addition, system summaries have been provided in the Appendices in Tables {{table:61816105-efd7-40ae-8269-d7c37ff04944}}  to {{table:ecd29ac1-2def-41b7-a252-2c7a9f4a490d}} .

The 2021 Lexical Complexity Prediction Task {{cite:306a4cd}}, referred to as LCP–2021, was also held at SemEval and attracted 58 teams across its 2 sub-tasks as shown within Table {{table:ecd29ac1-2def-41b7-a252-2c7a9f4a490d}}  (Appendices).


15,000 words labeled with varying degrees of complexity.

10,800 words and MWEs labeled with varying degrees of complexity.

11 non-native yet fluent English speakers.

Annotators were crowd sourced from the US, UK, and Australia. A median of 7 annotators labeled each word.

Used a dictomous six-point likert scale for annotation: very easy (very hard), moderately easy (moderately hard), and easy (hard).

Used a five-point likert scale for annotation. Scores were directly converted into continuous complex values. Words in context were taken from the Bible, biomedical articles, and europarl.

A Word-Complexity Lexicon and A Neural Readability Ranking Model for Lexical Simplification

Abstract: Current lexical simplification approaches rely heavily on heuristics and corpus level features that do not always align with human judgment. We create a human-rated word-complexity lexicon of 15,000 English words and propose a novel neural readability ranking model with a Gaussian-based feature vectorization layer that utilizes these human ratings to measure the complexity of any given word or phrase. Our model performs better than the state-of-the-art systems for different lexical simplification tasks and evaluation datasets. Additionally, we also produce SimplePPDB++, a lexical resource of over 10 million simplifying paraphrase rules, by applying our model to the Paraphrase Database (PPDB).

CompLex: A New Corpus for Lexical Complexity Prediction from Likert Scale Data

Abstract: Predicting which words are considered hard to understand for a given target population is a vital step in many NLP applications such as text simplification. This task is commonly referred to as Complex Word Identification (CWI). With a few exceptions, previous studies have approached the task as a binary classification task in which systems predict a complexity value (complex vs. non-complex) for a set of target words in a text. This choice is motivated by the fact that all CWI datasets compiled so far have been annotated using a binary annotation scheme. Our paper addresses this limitation by presenting the first English dataset for continuous lexical complexity prediction. We use a 5-point Likert scale scheme to annotate complex words in texts from three sources/domains: the Bible, Europarl, and biomedical texts. This resulted in a corpus of 9,476 sentences each annotated by around 7 annotators.

Datasets used for English complexity prediction research arranged in chronological order.

Additional English Datasets and Resources

The shared-tasks of CWI–2016 {{cite:a872c1b}}, CWI–2018 {{cite:c5d8783}}, and LCP–2021 {{cite:306a4cd}}, tested participating teams on three datasets that have since contributed significantly to LCP research (Section ). Nevertheless, these are not the only influential datasets that contain words with lexical complexity ratings. All of the current LCP datasets that deal with English and that are known to the authors' are provided in Table {{table:0b619965-ccfd-4f2b-8839-985631fe1507}}  located within the Appendices. Apart from the CWI–2016, CWI–2018, and the CompLex datasets already discussed within Section , the remaining datasets are introduced throughout the following sections (REF  to REF ).


Collection of nine different language understanding tasks

Question-Answering Tasks; max_query_length = 64, doc_stride = 128

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Abstract: For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems.

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Abstract: We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com

Summary of Benchmarks

Network architectures and Benchmarks. We evaluated X-Former on the benchmarks specified in Table {{table:1ee4d8fe-7db7-4e1b-b715-5035a01116da}}  using network configurations described in Table {{table:2f1d8fdf-ff25-4325-866b-a96664c04596}} . We note that for GLUE benchmark, the final results are an average over all the downstream tasks. In SQuAD benchmark, the doc_stride parameter denotes the stride length when a long document is split and max_query_length indicates the maximum number of tokens in a question {{cite:b2f7b8b}}.


Fuse, smoothing layers and skip connections

A Deep Multi-Level Network for Saliency Prediction

Abstract: This paper presents a novel deep architecture for saliency prediction. Current state of the art models for saliency prediction employ Fully Convolutional networks that perform a non-linear combination of features extracted from the last convolutional layer to predict saliency maps. We propose an architecture which, instead, combines features extracted at different levels of a Convolutional Neural Network (CNN). Our model is composed of three main blocks: a feature extraction CNN, a feature encoding network, that weights low and high level feature maps, and a prior learning network. We compare our solution with state of the art saliency models on two public benchmarks datasets. Results show that our model outperforms under all evaluation metrics on the SALICON dataset, which is currently the largest public dataset for saliency prediction, and achieves competitive results on the MIT300 benchmark.

Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model

Abstract: Data-driven saliency has recently gained a lot of attention thanks to the use of Convolutional Neural Networks for predicting gaze fixations. In this paper we go beyond standard approaches to saliency prediction, in which gaze maps are computed with a feed-forward network, and present a novel model which can predict accurate saliency maps by incorporating neural attentive mechanisms. The core of our solution is a Convolutional LSTM that focuses on the most salient regions of the input image to iteratively refine the predicted saliency map. Additionally, to tackle the center bias typical of human eye fixations, our model can learn a set of prior maps generated with Gaussian functions. We show, through an extensive evaluation, that the proposed architecture outperforms the current state of the art on public saliency prediction datasets. We further study the contribution of each key component to demonstrate their robustness on different scenarios.

TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection

Abstract: TASED-Net is a 3D fully-convolutional network architecture for video saliency detection. It consists of two building blocks: first, the encoder network extracts low-resolution spatiotemporal features from an input clip of several consecutive frames, and then the following prediction network decodes the encoded features spatially while aggregating all the temporal information. As a result, a single prediction map is produced from an input clip of multiple frames. Frame-wise saliency maps can be predicted by applying TASED-Net in a sliding-window fashion to a video. The proposed approach assumes that the saliency map of any frame can be predicted by considering a limited number of past frames. The results of our extensive experiments on video saliency detection validate this assumption and demonstrate that our fully-convolutional model with temporal aggregation method is effective. TASED-Net significantly outperforms previous state-of-the-art approaches on all three major large-scale datasets of video saliency detection: DHF1K, Hollywood2, and UCFSports. After analyzing the results qualitatively, we observe that our model is especially better at attending to salient moving objects.

Unified Image and Video Saliency Modeling

Abstract: Visual saliency modeling for images and videos is treated as two independent tasks in recent computer vision literature. While image saliency modeling is a well-studied problem and progress on benchmarks like SALICON and MIT300 is slowing, video saliency models have shown rapid gains on the recent DHF1K benchmark. Here, we take a step back and ask: Can image and video saliency modeling be approached via a unified model, with mutual benefit? We identify different sources of domain shift between image and video saliency data and between different video saliency datasets as a key challenge for effective joint modelling. To address this we propose four novel domain adaptation techniques - Domain-Adaptive Priors, Domain-Adaptive Fusion, Domain-Adaptive Smoothing and Bypass-RNN - in addition to an improved formulation of learned Gaussian priors. We integrate these techniques into a simple and lightweight encoder-RNN-decoder-style network, UNISAL, and train it jointly with image and video saliency data. We evaluate our method on the video saliency datasets DHF1K, Hollywood-2 and UCF-Sports, and the image saliency datasets SALICON and MIT300. With one set of parameters, UNISAL achieves state-of-the-art performance on all video saliency datasets and is on par with the state-of-the-art for image saliency datasets, despite faster runtime and a 5 to 20-fold smaller model size compared to all competing deep methods. We provide retrospective analyses and ablation studies which confirm the importance of the domain shift modeling. The code is available at https://github.com/rdroste/unisal

Architecture of four fully-supervised compared methods.

As listed in Table {{table:a244ef80-f678-4be2-a877-2c6bcd7df726}} , we report the architecture of the chosen compared methods in our experiments, including the encoders, decoders, and `learned priors' as an important component for comparisons. Here `learned prior' {{cite:e440480}}, {{cite:55aa968}}, {{cite:26668a2}} is a matrix whose size is {{formula:0265cf63-c11c-4c9f-b5a6-74226bbf60e9}}  that of the original image. All elements of `learned prior' are initialized to 1 and can be optimized during training. By dividing the final saliency map into a grid of non-overlapping cells that correspond to {{formula:1afaed6d-59fb-4a6a-847b-35fd4e23fef3}}  of the original image, `learned prior' assign weights to the attention predictions within each cell.


RoBERTa: A Robustly Optimized BERT Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

Muppet: Massive Multi-task Representations with Pre-Finetuning

Abstract: We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g. RoBERTa) and generation models (e.g. BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.

CamemBERT: a Tasty French Language Model

Abstract: Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models --in all languages except English-- very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

German's Next Language Model

Abstract: In this work we present the experiments which lead to the creation of our BERT and ELECTRA based German language models, GBERT and GELECTRA. By varying the input training data, model size, and the presence of Whole Word Masking (WWM) we were able to attain SoTA performance across a set of document classification and named entity recognition (NER) tasks for both models of base and large size. We adopt an evaluation driven approach in training these models and our results indicate that both adding more data and utilizing WWM improve model performance. By benchmarking against existing German models, we show that these models are the best German models to date. Our trained models will be made publicly available to the research community.

Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language

Abstract: The paper introduces methods of adaptation of multilingual masked language models for a specific language. Pre-trained bidirectional language models show state-of-the-art performance on a wide range of tasks including reading comprehension, natural language inference, and sentiment analysis. At the moment there are two alternative approaches to train such models: monolingual and multilingual. While language specific models show superior performance, multilingual models allow to perform a transfer from one language to another and solve tasks for different languages simultaneously. This work shows that transfer learning from a multilingual model to monolingual model results in significant growth of performance on such tasks as reading comprehension, paraphrase detection, and sentiment analysis. Furthermore, multilingual initialization of monolingual model substantially reduces training time. Pre-trained models for the Russian language are open sourced.

SheffieldVeraAI at SemEval-2023 Task 3: Mono and multilingual approaches for news genre, topic and persuasion technique classification

Abstract: This paper describes our approach for SemEval-2023 Task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. For Subtask 1 (News Genre), we propose an ensemble of fully trained and adapter mBERT models which was ranked joint-first for German, and had the highest mean rank of multi-language teams. For Subtask 2 (Framing), we achieved first place in 3 languages, and the best average rank across all the languages, by using two separate ensembles: a monolingual RoBERTa-MUPPETLARGE and an ensemble of XLM-RoBERTaLARGE with adapters and task adaptive pretraining. For Subtask 3 (Persuasion Techniques), we train a monolingual RoBERTa-Base model for English and a multilingual mBERT model for the remaining languages, which achieved top 10 for all languages, including 2nd for English. For each subtask, we compared monolingual and multilingual approaches, and considered class imbalance techniques.

Models used in each subtask. {{formula:44e0eeec-bffe-4824-92f5-3cf188899843}}  for final submission. Based on model selection of {{cite:7a5bb25}}.

The models used in each subtask are shown in Table {{table:420d5627-0163-4d1b-b070-a8e243fef981}} .


AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection

Abstract: We propose AffordanceNet, a new deep learning approach to simultaneously detect multiple objects and their affordances from RGB images. Our AffordanceNet has two branches: an object detection branch to localize and classify the object, and an affordance detection branch to assign each pixel in the object to its most probable affordance label. The proposed framework employs three key components for effectively handling the multiclass problem in the affordance mask: a sequence of deconvolutional layers, a robust resizing strategy, and a multi-task loss function. The experimental results on the public datasets show that our AffordanceNet outperforms recent state-of-the-art methods by a fair margin, while its end-to-end architecture allows the inference at the speed of 150ms per image. This makes our AffordanceNet well suitable for real-time robotic applications. Furthermore, we demonstrate the effectiveness of AffordanceNet in different testing environments and in real robotic applications. The source code is available at https://github.com/nqanh/affordance-net.

Phrase-Based Affordance Detection via Cyclic Bilateral Interaction

Abstract: Affordance detection, which refers to perceiving objects with potential action possibilities in images, is a challenging task since the possible affordance depends on the person's purpose in real-world application scenarios. The existing works mainly extract the inherent human–object dependencies from image/video to accommodate affordance properties that change dynamically. In this article, we explore to perceive affordances from a vision-language perspective, and consider the challenging phrase-based affordance detection task, i.e., given a set of phrases describing the potential actions, all the object regions in a scene with the same affordance should be detected. To this end, we propose a cyclic bilateral consistency enhancement network (CBCE-Net) to align language and vision features in a progressive manner. Specifically, the presented CBCE-Net consists of a mutual guided vision-language module that updates the common features of vision and language in a progressive manner, and a cyclic interaction module that facilitates the perception of possible interaction with objects in a cyclic manner. In addition, we extend the public purpose-driven affordance dataset (PAD) by annotating affordance categories with short phrases. The extensive contrastive experimental results demonstrate the superior performance of our method over nine typical methods from four relevant fields in terms of both objective metrics and visual quality.

Where2Act: From Pixels to Actions for Articulated 3D Objects

Abstract: One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment. In this paper, we take a step towards that long-term goal -- we extract highly localized actionable information related to elementary actions such as pushing or pulling for articulated objects with movable parts. For example, given a drawer, our network predicts that applying a pulling force on the handle opens the drawer. We propose, discuss, and evaluate novel network architectures that given image and depth data, predict the set of actions possible at each pixel, and the regions over articulated parts that are likely to move under the force. We propose a learning-from-interaction framework with an online data sampling strategy that allows us to train the network in simulation (SAPIEN) and generalizes across categories. Check the website for code and data release: https://cs.stanford.edu/~kaichun/where2act/

3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding

Abstract: The ability to understand the ways to interact with objects from visual cues, a.k.a. visual affordance, is essential to vision-guided robotic research. This involves categorizing, segmenting and reasoning of visual affordance. Relevant studies in 2D and 2.5D image domains have been made previously, however, a truly functional understanding of object affordance requires learning and prediction in the 3D physical domain, which is still absent in the community. In this work, we present a 3D AffordanceNet dataset, a benchmark of 23k shapes from 23 semantic object categories, annotated with 18 visual affordance categories. Based on this dataset, we provide three benchmarking tasks for evaluating visual affordance understanding, including full-shape, partial-view and rotation-invariant affordance estimations. Three state-of-the-art point cloud deep learning networks are evaluated on all tasks. In addition we also investigate a semi-supervised learning setup to explore the possibility to benefit from unlabeled data. Comprehensive results on our contributed dataset show the promise of visual affordance understanding as a valuable yet challenging benchmark.

Affordance Learning. Various works for serval tasks in the affordance community.

At present, the affordance area has made achievements in multiple tasks (Tab. {{table:51b6dc23-faae-4081-b75c-6ffb5497d896}} ). Some works are devoted to detecting the affordance region from 2D sources i.eimages and videos {{cite:134f51b}}, {{cite:051a509}}, {{cite:7946c8e}}, {{cite:632f5c6}}, {{cite:514aa9e}}, {{cite:8f3db08}}, and there are also some studies accomplish this task with the assistance of natural language {{cite:74c0ca2}}, {{cite:bdbf665}}, {{cite:140628e}}. They seek to detect or segment objects that afford the action in 2D data, but cannot perceive the specific parts corresponding to the affordance. Thus, another type of method brings a leap to locate the affordance region of objects from 2D sources {{cite:c223c9e}}, {{cite:c62a1ef}}, {{cite:893338d}}, {{cite:a5254c1}}. Since the affordance knowledge derived from 2D sources is hard to extrapolate specific interactive locations of objects in the 3D environment, some researchers explore grounding object affordance from 3D data {{cite:d3ba584}}, {{cite:365d037}}, {{cite:a6aa27f}}, {{cite:9a61e28}}. Such methods could map the connection between affordance and the 3D structure, but most of them detach the real interaction, and it may deter the generalization ability: for object structures that are not mapped to a specific affordance, it is usually hard to generalize through this type of method. T. Nagarajan et al{{cite:9a61e28}} give a fresh mind, taking the reinforcement learning to make agents actively interact with the 3D synthetic scenarios, while it requires repeated attempts by the agents in a given search space, and is time-consuming. In robotics, some researchers exploit affordances provided by objects for manipulation. A series of studies explore manipulation affordance for the articulated 3D objects {{cite:f581c67}}, {{cite:ce699cb}}, {{cite:63b2fd7}} and achieve considerable results. Serval methods utilize the 2.5D data e.gRGB-D image to understand the object affordance and serve the manipulation task {{cite:ec15b20}}, {{cite:e3ceca4}}, {{cite:e5a4ba8}}, {{cite:c319622}}, the image and depth information are corresponding in spatial and need to be collected in the same scene. Our task focus on grounding 3D object affordance from 2D interactions. In which the 2D interactions provide direct clues to excavate object affordance efficiently and make the affordance could generalize to some unseen structures. And the paired 2D-3D data is collected from different sources, freeing the constraints on spatial correspondence.


AI service placement in mobile edge intelligence

Multi-user collaborative DNN partitioning

Total time and energy consumption/Service placement decision, local CPU frequencies, uplink bandwidth, edge CPU frequency

Execution latency/Partitioning, computation resources

Fully utilize scarce wireless spectrum and edge computing resources in provisioning AIGC services

Providing insights for partitioning  generative AI models under edge-mobile collaboration

Optimizing AI Service Placement and Resource Allocation in Mobile Edge Intelligence Systems

Abstract: Leveraging recent advances on mobile edge computing (MEC), edge intelligence has emerged as a promising paradigm to support mobile artificial intelligence (AI) applications at the network edge. In this paper, we consider the AI service placement problem in a multi-user MEC system, where the access point (AP) places the most up-to-date AI program at user devices to enable local computing/task execution at the user side. To fully utilize the stringent wireless spectrum and edge computing resources, the AP sends the AI service program to a user only when enabling local computing at the user yields a better system performance. We formulate a mixed-integer non-linear programming (MINLP) problem to minimize the total computation time and energy consumption of all users by jointly optimizing the service placement (i.e., which users to receive the program) and resource allocation (on local CPU frequencies, uplink bandwidth, and edge CPU frequency). To tackle the MINLP problem, we derive analytical expressions to calculate the optimal resource allocation decisions with low complexity. This allows us to efficiently obtain the optimal service placement solution by search-based algorithms such as meta-heuristic or greedy search algorithms. To enhance the algorithm scalability in large-sized networks, we further propose an ADMM (alternating direction method of multipliers) based method to decompose the optimization problem into parallel tractable MINLP subproblems. The ADMM method eliminates the need of searching in a high-dimensional space for service placement decisions and thus has a low computational complexity that grows linearly with the number of users. Simulation results show that the proposed algorithms perform extremely close to the optimum and significantly outperform the other representative benchmark algorithms.

Joint Multi-User DNN Partitioning and Computational Resource Allocation for Collaborative Edge Intelligence

Abstract: Mobile Edge Computing (MEC) has emerged as a promising supporting architecture providing a variety of resources to the network edge, thus acting as an enabler for edge intelligence services empowering massive mobile and Internet of Things (IoT) devices with AI capability. With the assistance of edge servers, user equipments (UEs) are able to run deep neural network (DNN) based AI applications, which are generally resource-hungry and compute-intensive, such that an individual UE can hardly afford by itself in real time. However the resources in each individual edge server are typically limited. Therefore, any resource optimization involving edge servers is by nature a resource-constrained optimization problem and needs to be tackled in such realistic context. Motivated by this observation, we investigate the optimization problem of DNN partitioning (an emerging DNN offloading scheme) in a realistic multi-user resource-constrained condition that rarely considered in previous works. Despite the extremely large solution space, we reveal several properties of this specific optimization problem of joint multi-UE DNN partitioning and computational resource allocation. We propose an algorithm called Iterative Alternating Optimization (IAO) that can achieve the optimal solution in polynomial time. In addition, we present rigorous theoretic analysis of our algorithm in terms of time complexity and performance under realistic estimation error. Moreover, we build a prototype that implements our framework and conduct extensive experiments using realistic DNN models, whose results demonstrate its effectiveness and efficiency.

Summary of scenarios, problems, benefits/challenges, and mathematical tools of edge resource allocation.

AIGC service provisioning based on edge intelligence is computationally and communication-intensive for resource-constrained edge servers and mobile devices {{cite:541638a}}, {{cite:ada72f4}}. Specifically, AIGC users send service allocation requests to edge services. Upon receiving these AIGC requests, edge servers perform the AIGC tasks and deliver the output to users {{cite:68b13fc}}. During this AIGC service provisioning interaction, model accuracy and resource consumption are the most common metrics. Consequently, significant efforts are being made to coordinate mobile devices and edge servers for deploying generative AI at mobile edge networks. As summarized in Table {{table:b3fba7b6-ae8c-444b-ac42-489a54ef36b1}} , several Key Performance Indicators (KPIs) for edge resource allocation in AIGC networks are presented below.


A Theory of Non‐Bayesian Social Learning

Abstract: This paper studies the behavioral foundations of nonâ€ Bayesian models of learning over social networks and develops a taxonomy of conditions for information aggregation in a general framework. As our main behavioral assumption, we postulate that agents follow social learning rules that satisfy â€œimperfect recall,â€ according to which they treat the current beliefs of their neighbors as sufficient statistics for the entire history of their observations. We augment this assumption with various restrictions on how agents process the information provided by their neighbors and obtain representation theorems for the corresponding learning rules (including the canonical model of DeGroot). We then obtain general longâ€ run learning results that are not tied to the learning rules' specific functional forms, thus identifying the fundamental forces that lead to learning, nonâ€ learning, and mislearning in social networks. Our results illustrate that, in the presence of imperfect recall, longâ€ run aggregation of information is closely linked to (i) the rate at which agents discount their neighbors' information over time, (ii) the curvature of agents' social learning rules, and (iii) whether their initial tendencies are amplified or moderated as a result of social interactions.

Learning in Repeated Interactions on Networks

Abstract: We study how long-lived, rational agents learn in a social network. In every period, after observing the past actions of his neighbors, each agent receives a private signal, and chooses an action whose payoff depends only on the state. Since equilibrium actions depend on higher order beliefs, it is difficult to characterize behavior. Nevertheless, we show that regardless of the size and shape of the network, the utility function, and the patience of the agents, the speed of learning in any equilibrium is bounded from above by a constant that only depends on the private signal distribution.

Modelling decisions in selected social network learning literature.

Social network learning concerns agents who try to optimize an unknown objective function by choosing an action, and that are learning via private signals and/or signals from other agents. The modelling decisions relate to the degree of rationality in learning, the communication between the agents, and whether or not the private signals of the agents are conditional on the actions they take. Table {{table:b32d373e-eae5-4c5a-b4fd-a64c5cdec195}}  summarises the modelling decisions of the papers {{cite:40a2427}}, {{cite:74fed24}}, {{cite:cfe55b8}}, {{cite:1ee5cbd}}, all of them considering a population of agents in the social network learning framework. The second last column headed “Communication” refers to what the agents share with one another. This could be their private learning signal per round, their belief distribution, or simply the actions they take. The last column headed “Signals” refers to whether the reception of private signals by the agents is independent of or conditional on the actions the agents take.


Unsupervised Keyphrase Extraction with Multipartite Graphs

Abstract: We propose an unsupervised keyphrase extraction model that encodes topical information within a multipartite graph structure. Our model represents keyphrase candidates and topics in a single graph and exploits their mutually reinforcing relationship to improve candidate ranking. We further introduce a novel mechanism to incorporate keyphrase selection preferences into the model. Experiments conducted on three widely used datasets show significant improvements over state-of-the-art graph-based models.

KEA: Practical Automatic Keyphrase Extraction

Abstract: Keyphrases provide semantic metadata that summarize and characterize documents. This paper describes Kea, an algorithm for automatically extracting keyphrases from text. Kea identifies candidate keyphrases using lexical methods, calculates feature values for each candidate, and uses a machine-learning algorithm to predict which candidates are good keyphrases. The machine learning scheme first builds a prediction model using training documents with known keyphrases, and then uses the model to find keyphrases in new documents. We use a large test corpus to evaluate Kea's effectiveness in terms of how many author-assigned keyphrases are correctly identified. The system is simple, robust, and publicly available.

Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study

Abstract: Neural models that do not rely on pre-training have excelled in the keyphrase generation task with large annotated datasets. Meanwhile, new approaches have incorporated pre-trained language models (PLMs) for their data efficiency. However, there lacks a systematic study of how the two types of approaches compare and how different design choices can affect the performance of PLM-based models. To fill in this knowledge gap and facilitate a more informed use of PLMs for keyphrase extraction and keyphrase generation, we present an in-depth empirical study. Formulating keyphrase extraction as sequence labeling and keyphrase generation as sequence-to-sequence generation, we perform extensive experiments in three domains. After showing that PLMs have competitive high-resource performance and state-of-the-art low-resource performance, we investigate important design choices including in-domain PLMs, PLMs with different pre-training objectives, using PLMs with a parameter budget, and different formulations for present keyphrases. Further results show that (1) in-domain BERT-like PLMs can be used to build strong and data-efficient keyphrase generation models; (2) with a fixed parameter budget, prioritizing model depth over width and allocating more layers in the encoder leads to better encoder-decoder models; and (3) introducing four in-domain PLMs, we achieve a competitive performance in the news domain and the state-of-the-art performance in the scientific domain.

Neural Keyphrase Generation via Reinforcement Learning with Adaptive Rewards

Abstract: Generating keyphrases that summarize the main points of a document is a fundamental task in natural language processing. Although existing generative models are capable of predicting multiple keyphrases for an input document as well as determining the number of keyphrases to generate, they still suffer from the problem of generating too few keyphrases. To address this problem, we propose a reinforcement learning (RL) approach for keyphrase generation, with an adaptive reward function that encourages a model to generate both sufficient and accurate keyphrases. Furthermore, we introduce a new evaluation method that incorporates name variations of the ground-truth keyphrases using the Wikipedia knowledge base. Thus, our evaluation method can more robustly evaluate the quality of predicted keyphrases. Extensive experiments on five real-world datasets of different scales demonstrate that our RL approach consistently and significantly improves the performance of the state-of-the-art generative models with both conventional and new evaluation methods.

Exclusive Hierarchical Decoding for Deep Keyphrase Generation

Abstract: Keyphrase generation (KG) aims to summarize the main ideas of a document into a set of keyphrases. A new setting is recently introduced into this problem, in which, given a document, the model needs to predict a set of keyphrases and simultaneously determine the appropriate number of keyphrases to produce. Previous work in this setting employs a sequential decoding process to generate keyphrases. However, such a decoding method ignores the intrinsic hierarchical compositionality existing in the keyphrase set of a document. Moreover, previous work tends to generate duplicated keyphrases, which wastes time and computing resources. To overcome these limitations, we propose an exclusive hierarchical decoding framework that includes a hierarchical decoding process and either a soft or a hard exclusion mechanism. The hierarchical decoding process is to explicitly model the hierarchical compositionality of a keyphrase set. Both the soft and the hard exclusion mechanisms keep track of previously-predicted keyphrases within a window size to enhance the diversity of the generated keyphrases. Extensive experiments on multiple KG benchmark datasets demonstrate the effectiveness of our method to generate less duplicated and more accurate keyphrases.

One2Set: Generating Diverse Keyphrases as a Set

Abstract: Recently, the sequence-to-sequence models have made remarkable progress on the task of keyphrase generation (KG) by concatenating multiple keyphrases in a predefined order as a target sequence during training. However, the keyphrases are inherently an unordered set rather than an ordered sequence. Imposing a predefined order will introduce wrong bias during training, which can highly penalize shifts in the order between keyphrases. In this work, we propose a new training paradigm One2Set without predefining an order to concatenate the keyphrases. To fit this paradigm, we propose a novel model that utilizes a fixed set of learned control codes as conditions to generate a set of keyphrases in parallel. To solve the problem that there is no correspondence between each prediction and target during training, we propose a K-step label assignment mechanism via bipartite matching, which greatly increases the diversity and reduces the repetition rate of generated keyphrases. The experimental results on multiple benchmarks demonstrate that our approach significantly outperforms the state-of-the-art methods.

Learning Rich Representation of Keyphrases from Text

Abstract: In this work, we explore how to train task-specific language models aimed towards learning rich representation of keyphrases from text documents. We experiment with different masking strategies for pre-training transformer language models (LMs) in discriminative as well as generative settings. In the discriminative setting, we introduce a new pre-training objective - Keyphrase Boundary Infilling with Replacement (KBIR), showing large gains in performance (upto 8.16 points in F1) over SOTA, when the LM pre-trained using KBIR is fine-tuned for the task of keyphrase extraction. In the generative setting, we introduce a new pre-training setup for BART - KeyBART, that reproduces the keyphrases related to the input text in the CatSeq format, instead of the denoised original input. This also led to gains in performance (upto 4.33 points in F1@M) over SOTA for keyphrase generation. Additionally, we also fine-tune the pre-trained language models on named entity recognition (NER), question answering (QA), relation extraction (RE), abstractive summarization and achieve comparable performance with that of the SOTA, showing that learning rich representation of keyphrases is indeed beneficial for many other fundamental NLP tasks.

KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation

Abstract: Despite the significant advancements in keyphrase extraction and keyphrase generation methods, the predominant approach for evaluation mainly relies on exact matching with human references. This scheme fails to recognize systems that generate keyphrases semantically equivalent to the references or diverse keyphrases that carry practical utility. To better assess the capability of keyphrase systems, we propose KPEval, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. For each aspect, we design semantic-based metrics to reflect the evaluation objectives. Meta-evaluation studies demonstrate that our evaluation strategy correlates better with human preferences compared to a range of previously proposed metrics. Using KPEval, we re-evaluate 21 keyphrase systems and discover that (1) established model comparison results have blind-spots especially when considering reference-free evaluation; (2) large language models are underestimated by prior evaluation works; and (3) there is no single best model that can excel in all the aspects.

Descriptions of the studied models. "KPE" = "Keyphrase Extraction". "KPG" = "Keyphrase Generation". "PLM" = "Pre-trained Language Model". "TAPT" = "Task-adaptive Pre-training".

In this section, we outline the keyphrase systems and datasets we will study in this paper and discuss their related work. We select 18 models from three main categories, as summarized in Table {{table:55a499cc-e638-4a6a-ae1a-d7987b426322}}  in appendix. Note that our goal is to include strong systems of diverse types rather than exhaustively benchmarking all available systems.

Detailed information of the considered keyphrase systems

We summarize the properties of all the considered keyphrase systems in Table {{table:55a499cc-e638-4a6a-ae1a-d7987b426322}} . We benchmark 7 keyphrase extraction models and 11 keyphrase generation models. Among the keyphrase generation models, 5 models are trained from scratch, 4 models are fine-tuned from pre-trained language models, and 2 are prompting GPT-3.5 either zero-shot or with with examples.


Personalized Federated Learning with Communication Compression

Abstract: In contrast to training traditional machine learning (ML) models in data centers, federated learning (FL) trains ML models over local datasets contained on resource-constrained heterogeneous edge devices. Existing FL algorithms aim to learn a single global model for all participating devices, which may not be helpful to all devices participating in the training due to the heterogeneity of the data across the devices. Recently, Hanzely and Richt\'{a}rik (2020) proposed a new formulation for training personalized FL models aimed at balancing the trade-off between the traditional global model and the local models that could be trained by individual devices using their private data only. They derived a new algorithm, called Loopless Gradient Descent (L2GD), to solve it and showed that this algorithms leads to improved communication complexity guarantees in regimes when more personalization is required. In this paper, we equip their L2GD algorithm with a bidirectional compression mechanism to further reduce the communication bottleneck between the local devices and the server. Unlike other compression-based algorithms used in the FL-setting, our compressed L2GD algorithm operates on a probabilistic communication protocol, where communication does not happen on a fixed schedule. Moreover, our compressed L2GD algorithm maintains a similar convergence rate as vanilla SGD without compression. To empirically validate the efficiency of our algorithm, we perform diverse numerical experiments on both convex and non-convex problems and using various compression techniques.

Architecture Agnostic Federated Learning for Neural Networks

Abstract: With growing concerns regarding data privacy and rapid increase in data volume, Federated Learning(FL) has become an important learning paradigm. However, jointly learning a deep neural network model in a FL setting proves to be a non-trivial task because of the complexities associated with the neural networks, such as varied architectures across clients, permutation invariance of the neurons, and presence of non-linear transformations in each layer. This work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN) framework that allows each client to build a personalised model without enforcing a common architecture across clients. This allows each client to optimize with respect to local data and compute constraints, while still benefiting from the learnings of other (potentially more powerful) clients. The key idea of FedHeNN is to use the instance-level representations obtained from peer clients to guide the simultaneous training on each client. The extensive experimental results demonstrate that the FedHeNN framework is capable of learning better performing models on clients in both the settings of homogeneous and heterogeneous architectures across clients.

Exploiting Shared Representations for Personalized Federated Learning

Abstract: Deep neural networks have shown the ability to extract universal feature representations from data such as images and text that have been useful for a variety of learning tasks. However, the fruits of representation learning have yet to be fully-realized in federated settings. Although data in federated settings is often non-i.i.d. across clients, the success of centralized deep learning suggests that data often shares a global feature representation, while the statistical heterogeneity across clients or tasks is concentrated in the labels. Based on this intuition, we propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. We prove that this method obtains linear convergence to the ground-truth representation with near-optimal sample complexity in a linear setting, demonstrating that it can efficiently reduce the problem dimension for each client. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions, for example in meta-learning and multi-task learning. Further, extensive experimental results show the empirical improvement of our method over alternative personalized federated learning approaches in federated environments with heterogeneous data.

Throughput-Optimal Topology Design for Cross-Silo Federated Learning

Abstract: Federated learning usually employs a client-server architecture where an orchestrator iteratively aggregates model updates from remote clients and pushes them back a refined model. This approach may be inefficient in cross-silo settings, as close-by data silos with high-speed access links may exchange information faster than with the orchestrator, and the orchestrator may become a communication bottleneck. In this paper we define the problem of topology design for cross-silo federated learning using the theory of max-plus linear systems to compute the system throughput---number of communication rounds per time unit. We also propose practical algorithms that, under the knowledge of measurable network characteristics, find a topology with the largest throughput or with provable throughput guarantees. In realistic Internet networks with 10 Gbps access links for silos, our algorithms speed up training by a factor 9 and 1.5 in comparison to the master-slave architecture and to state-of-the-art MATCHA, respectively. Speedups are even larger with slower access links.

Ensemble Distillation for Robust Model Fusion in Federated Learning

Abstract: Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100, ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the server model can be trained much faster, requiring fewer communication rounds than any existing FL technique so far.

Federated Learning with Fair Averaging

Abstract: Fairness has emerged as a critical problem in federated learning (FL). In this work, we identify a cause of unfairness in FL -- conflicting gradients with large differences in the magnitudes. To address this issue, we propose the federated fair averaging (FedFV) algorithm to mitigate potential conflicts among clients before averaging their gradients. We first use the cosine similarity to detect gradient conflicts, and then iteratively eliminate such conflicts by modifying both the direction and the magnitude of the gradients. We further show the theoretical foundation of FedFV to mitigate the issue conflicting gradients and converge to Pareto stationary solutions. Extensive experiments on a suite of federated datasets confirm that FedFV compares favorably against state-of-the-art methods in terms of fairness, accuracy and efficiency. The source code is available at https://github.com/WwZzz/easyFL.

Communication-Efficient Learning of Deep Networks from Decentralized Data

Abstract:
Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning.
We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.

Federated Learning with Matched Averaging

Abstract: Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud. We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs. FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) with similar feature extraction signatures. Our experiments indicate that FedMA not only outperforms popular state-of-the-art federated learning algorithms on deep CNN and LSTM architectures trained on real world datasets, but also reduces the overall communication burden.

Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System

Abstract: The increasing interest in user privacy is leading to new privacy preserving machine learning paradigms. In the Federated Learning paradigm, a master machine learning model is distributed to user clients, the clients use their locally stored data and model for both inference and calculating model updates. The model updates are sent back and aggregated on the server to update the master model then redistributed to the clients. In this paradigm, the user data never leaves the client, greatly enhancing the user' privacy, in contrast to the traditional paradigm of collecting, storing and processing user data on a backend server beyond the user's control. In this paper we introduce, as far as we are aware, the first federated implementation of a Collaborative Filter. The federated updates to the model are based on a stochastic gradient approach. As a classical case study in machine learning, we explore a personalized recommendation system based on users' implicit feedback and demonstrate the method's applicability to both the MovieLens and an in-house dataset. Empirical validation confirms a collaborative filter can be federated without a loss of accuracy compared to a standard implementation, hence enhancing the user's privacy in a widely used recommender application while maintaining recommender performance.

The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems

Abstract: This paper presents and characterizes an Open Application Repository for Federated Learning (OARF), a benchmark suite for federated machine learning systems. Previously available benchmarks for federated learning have focused mainly on synthetic datasets and use a limited number of applications. OARF mimics more realistic application scenarios with publicly available data sets as different data silos in image, text and structured data. Our characterization shows that the benchmark suite is diverse in data size, distribution, feature distribution and learning task complexity. The extensive evaluations with reference implementations show the future research opportunities for important aspects of federated learning systems. We have developed reference implementations, and evaluated the important aspects of federated learning, including model accuracy, communication cost, throughput and convergence time. Through these evaluations, we discovered some interesting findings such as federated learning can effectively increase end-to-end throughput.

Agnostic Federated Learning

Abstract: A key learning scenario in large-scale applications is that of federated learning, where a centralized model is trained based on data originating from a large number of clients. We argue that, with the existing training and inference, federated models can be biased towards different clients. Instead, we propose a new framework of agnostic federated learning, where the centralized model is optimized for any target distribution formed by a mixture of the client distributions. We further show that this framework naturally yields a notion of fairness. We present data-dependent Rademacher complexity guarantees for learning with this objective, which guide the definition of an algorithm for agnostic federated learning. We also give a fast stochastic optimization algorithm for solving the corresponding optimization problem, for which we prove convergence bounds, assuming a convex loss function and hypothesis set. We further empirically demonstrate the benefits of our approach in several datasets. Beyond federated learning, our framework and algorithm can be of interest to other learning scenarios such as cloud computing, domain adaptation, drifting, and other contexts where the training and test distributions do not coincide.

Federated Multi-Task Learning

Abstract: Federated learning poses new statistical and systems challenges in training machine learning models over distributed networks of devices. In this work, we show that multi-task learning is naturally suited to handle the statistical challenges of this setting, and propose a novel systems-aware optimization method, MOCHA, that is robust to practical systems issues. Our method and theory for the first time consider issues of high communication cost, stragglers, and fault tolerance for distributed multi-task learning. The resulting method achieves significant speedups compared to alternatives in the federated setting, as we demonstrate through simulations on real-world federated datasets.

Comparison between existing works on FL. We denote Cryptographic Methods as CM, Differential Privacy as DP, Linear Models as LM, Neural Networks as NN, and Decision Tree as DT.

Different algorithms or methods are leveraged by different practitioners to construct one or multiple types of FL architectures such as Linear Models (LM), Decision Trees (DT) and Neural Network (NN), etc. Apparently, most FL settings have horizontal data partitioning. Table {{table:b1d7bdf0-7315-4279-9a3c-51d6fc720a31}}  summarizes several popular researches.


Four fundamental tradeoffs for green radios have been discovered.

Energy efficient vehicular communications have been proposed.

Fundamental Tradeoffs on Green Wireless Networks

Abstract: Traditional design of mobile wireless networks mainly focuses on ubiquitous access and large capacity. However, as energy saving and environmental protection become a global demand and inevitable trend, wireless researchers and engineers need to shift their focus to energy-efficiency oriented design, that is, green radio. In this paper, we propose a framework for green radio research and integrate the fundamental issues that are currently scattered. The skeleton of the framework consists of four fundamental tradeoffs: deployment efficiency - energy efficiency tradeoff, spectrum efficiency - energy efficiency tradeoff, bandwidth - power tradeoff, and delay - power tradeoff. With the help of the four fundamental tradeoffs, we demonstrate that key network performance/cost indicators are all stringed together.

Vehicular Communications: A Physical Layer Perspective

Abstract: Vehicular communications have attracted more and more attention recently from both industry and academia due to their strong potential to enhance road safety, improve traffic efficiency, and provide rich on-board information and entertainment services. In this paper, we discuss fundamental physical layer issues that enable efficient vehicular communications and present a comprehensive overview of the state-of-the-art research. We first introduce vehicular channel characteristics and modeling, which are the key underlying features differentiating vehicular communications from other types of wireless systems. We then present schemes to estimate the time-varying vehicular channels and various modulation techniques to deal with high-mobility channels. After reviewing resource allocation for vehicular communications, we discuss the potential to enable vehicular communications over the millimeter wave bands. Finally, we identify the challenges and opportunities associated with vehicular communications.

List of Milestones for Green Radios during the First 20 Years.

There has been tremendous achievement in green radios in the past 20 years. In Table {{table:593ea5f6-ad8a-4c1a-934f-c222dd39a613}} , we summarize the milestones of green radios during the first 20 years. In addition to that, part of research efforts have also been contributed to the backbone infrastructure, also known as core networks of wireless communications, including network routers, optical cross-connect, and transponders. Typical green solutions for them include the energy efficient core network architecture design {{cite:89e719f}}, efficient routing and packet forwarding protocol design {{cite:762c15d}}, and selectively turning off idle elements {{cite:8389d66}}. Since the major energy consumption of “radios” is in the access network and terminal sides, we refer interested readers to reference {{cite:c84765e}} for more information on the energy efficient design for core networks.


Generating fully labeled, dynamic, and photo-realistic proxy virtual world, with a focus on objects of interest, e.g. cars.

Generating synthetic data for semantic segmentation of outdoor scenes, for recognizing aspects such as roads, buildings, cars, people, lights etc.

Creating synthetic images to predict number of tomatoes in the images.

Synthetic data for 2D bounding box car detection.

Generating 3D scenes of visually realistic houses, ranging from single-room studios to multi-storied houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layout, for teaching an agent to navigate in an unseen 3D environment.

Generating scenes for teaching an artificial agent to execute tasks in a simulated household environment.

a) Generating data for semantic segmentation of aerial views of roadways. b) Simulating urban scenes for object detection in urban car driving.

VirtualWorlds as Proxy for Multi-object Tracking Analysis

Abstract: Modern computer vision algorithms typically require expensive data acquisition and accurate manual labeling. In this work, we instead leverage the recent progress in computer graphics to generate fully labeled, dynamic, and photo-realistic proxy virtual worlds. We propose an efficient real-to-virtual world cloning method, and validate our approach by building and publicly releasing a new video dataset, called "Virtual KITTI", automatically labeled with accurate ground truth for object detection, tracking, scene and instance segmentation, depth, and optical flow. We provide quantitative experimental evidence suggesting that (i) modern deep learning algorithms pre-trained on real data behave similarly in real and virtual worlds, and (ii) pre-training on virtual data improves performance. As the gap between real and virtual worlds is small, virtual worlds enable measuring the impact of various weather and imaging conditions on recognition performance, all other things being equal. We show these factors may affect drastically otherwise high-performing deep models for tracking.

Playing for Data: Ground Truth from Computer Games

Abstract: Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set.

Deep Count: Fruit Counting Based on Deep Simulated Learning

Abstract: Recent years have witnessed significant advancement in computer vision research based on deep learning. Success of these tasks largely depends on the availability of a large amount of training samples. Labeling the training samples is an expensive process. In this paper, we present a simulated deep convolutional neural network for yield estimation. Knowing the exact number of fruits, flowers, and trees helps farmers to make better decisions on cultivation practices, plant disease prevention, and the size of harvest labor force. The current practice of yield estimation based on the manual counting of fruits or flowers by workers is a very time consuming and expensive process and it is not practical for big fields. Automatic yield estimation based on robotic agriculture provides a viable solution in this regard. Our network is trained entirely on synthetic data and tested on real data. To capture features on multiple scales, we used a modified version of the Inception-ResNet architecture. Our algorithm counts efficiently even if fruits are under shadow, occluded by foliage, branches, or if there is some degree of overlap amongst fruits. Experimental results show a 91% average test accuracy on real images and 93% on synthetic images.

Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data

Abstract: We present structured domain randomization (SDR), a variant of domain randomization (DR) that takes into account the structure of the scene in order to add context to the generated data. In contrast to DR, which places objects and distractors randomly according to a uniform probability distribution, SDR places objects and distractors randomly according to probability distributions that arise from the specific problem at hand. In this manner, SDR-generated imagery enables the neural network to take the context around an object into consideration during detection. We demonstrate the power of SDR for the problem of 2D bounding box car detection, achieving competitive results on real data after training only on synthetic data. On the KITTI easy, moderate, and hard tasks, we show that SDR outperforms other approaches to generating synthetic data (VKITTI, Sim 200k, or DR), as well as real data collected in a different domain (BDD100K). Moreover, synthetic SDR data combined with real KITTI data outperforms real KITTI data alone.11Video is at http://youtu.be/1WdjWJYx9AY

Building Generalizable Agents with a Realistic and Rich 3D Environment

Abstract: Teaching an agent to navigate in an unseen 3D environment is a challenging task, even in the event of simulated environments. To generalize to unseen environments, an agent needs to be robust to low-level variations (e.g. color, texture, object changes), and also high-level variations (e.g. layout changes of the environment). To improve overall generalization, all types of variations in the environment have to be taken under consideration via different level of data augmentation steps. To this end, we propose House3D, a rich, extensible and efficient environment that contains 45,622 human-designed 3D scenes of visually realistic houses, ranging from single-room studios to multi-storied houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layouts, based on the SUNCG dataset (Song et.al.). The diversity in House3D opens the door towards scene-level augmentation, while the label-rich nature of House3D enables us to inject pixel- & task-level augmentations such as domain randomization (Toubin et. al.) and multi-task training. Using a subset of houses in House3D, we show that reinforcement learning agents trained with an enhancement of different levels of augmentations perform much better in unseen environments than our baselines with raw RGB input by over 8% in terms of navigation success rate. House3D is publicly available at http://github.com/facebookresearch/House3D.

VirtualHome: Simulating Household Activities via Programs

Abstract: In this paper, we are interested in modeling complex activities that occur in a typical household. We propose to use programs, i.e., sequences of atomic actions and interactions, as a high level representation of complex tasks. Programs are interesting because they provide a non-ambiguous representation of a task, and allow agents to execute them. However, nowadays, there is no database providing this type of information. Towards this goal, we first crowd-source programs for a variety of activities that happen in people's homes, via a game-like interface used for teaching kids how to code. Using the collected dataset, we show how we can learn to extract programs directly from natural language descriptions or from videos. We then implement the most common atomic (inter)actions in the Unity3D game engine, and use our programs to "drive" an artificial agent to execute tasks in a simulated household environment. Our VirtualHome simulator allows us to create a large activity video dataset with rich ground-truth, enabling training and testing of video understanding models. We further showcase examples of our agent performing tasks in our VirtualHome based on language descriptions.

Meta-Sim: Learning to Generate Synthetic Datasets

Abstract: Training models to high-end performance requires availability of large labeled datasets, which are expensive to get. The goal of our work is to automatically synthesize labeled datasets that are relevant for a downstream task. We propose Meta-Sim, which learns a generative model of synthetic scenes, and obtain images as well as its corresponding ground-truth via a graphics engine. We parametrize our dataset generator with a neural network, which learns to modify attributes of scene graphs obtained from probabilistic scene grammars, so as to minimize the distribution gap between its rendered outputs and target data. If the real dataset comes with a small labeled validation set, we additionally aim to optimize a meta-objective, i.e. downstream task performance. Experiments show that the proposed method can greatly improve content generation quality over a human-engineered probabilistic scene grammar, both qualitatively and quantitatively as measured by performance on a downstream task.

Related work in generative data for training DL models.

Table {{table:a573a34e-76fb-45fa-8f61-7c8112958b92}}  lists related work in the field of generating training data to train DL models. The year of publication for every paper reveals how modern this technique is. Please note that we avoided adding details about performance metrics and evaluation results for each paper, because each author used different metrics and experimented on different real-world datasets for testing. However, the general conclusion in all papers was that the performance according to the metric(s) used, was better than baseline (i.e. datasets not enhanced with synthetic data) or state-of-art related work.

From Table {{table:a573a34e-76fb-45fa-8f61-7c8112958b92}} , it is evident that related work has not entered yet the domain of UAV-based imagery analysis. The only exception is Meta-Sim {{cite:ca75a3a}}, which tries to learn a generative model of synthetic scenes automatically, via probabilistic scene grammars, and then it obtains images and their corresponding ground-truth via a graphics engine. Meta-Sim validates this idea addressing the problem of semantic segmentation of simulated aerial views of simple roadways. Beyond this work, to our knowledge, no other work has focused yet on generative data-based approaches for UAV-based imaging-related applications.


tensor-train decomposition for high-dim integration

statistical tensor completion to predict variation pattern

Enabling High-Dimensional Hierarchical Uncertainty Quantification by ANOVA and Tensor-Train Decomposition

Abstract: Hierarchical uncertainty quantification can reduce the computational cost of stochastic circuit simulation by employing spectral methods at different levels. This paper presents an efficient framework to simulate hierarchically some challenging stochastic circuits/systems that include high-dimensional subsystems. Due to the high parameter dimensionality, it is challenging to both extract surrogate models at the low level of the design hierarchy and to handle them in the high-level simulation. In this paper, we develop an efficient ANOVA-based stochastic circuit/MEMS simulator to extract efficiently the surrogate models at the low level. In order to avoid the curse of dimensionality, we employ tensor-train decomposition at the high level to construct the basis functions and Gauss quadrature points. As a demonstration, we verify our algorithm on a stochastic oscillator with four MEMS capacitors and 184 random parameters. This challenging example is simulated efficiently by our simulator at the cost of only 10 minutes in MATLAB on a regular personal computer.

Prediction of multi-dimensional spatial variation data via Bayesian tensor completion

Abstract: This paper presents a multi-dimensional computational method to predict the spatial variation data inside and across multiple dies of a wafer. This technique is based on tensor computation. A tensor is a high-dimensional generalization of a matrix or a vector. By exploiting the hidden low-rank property of a high-dimensional data array, the large amount of unknown variation testing data may be predicted from a few random measurement samples. The tensor rank, which decides the complexity of a tensor representation, is decided by an available variational Bayesian approach. Our approach is validated by a practical chip testing data set, and it can be easily generalized to characterize the process variations of multiple wafers. Our approach is more efficient than the previous virtual probe techniques in terms of memory and computational cost when handling high-dimensional chip testing data.

Application of tensors in uncertainty propagation and variability Modeling

Tensor techniques can be employed to achieve both goals. Representative tensor techniques for the first goal include {{cite:d175db5}}, {{cite:7429265}}, {{cite:0b9ce22}}. In this paper, we focus on tensor techniques for uncertainty and variability analysis as summarized in Table {{table:89f92841-a216-4575-b87e-e20b87b59894}} , and we elaborate their key ideas below.


Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

Abstract: Deep convolutional networks have achieved great success for visual recognition in still images. However, for action recognition in videos, the advantage over traditional methods is not so evident. This paper aims to discover the principles to design effective ConvNet architectures for action recognition in videos and learn these models given limited training samples. Our first contribution is temporal segment network (TSN), a novel framework for video-based action recognition. which is based on the idea of long-range temporal structure modeling. It combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. The other contribution is our study on a series of good practices in learning ConvNets on video data with the help of temporal segment network. Our approach obtains the state-the-of-art performance on the datasets of HMDB51 ( $ 69.4\% $) and UCF101 ($ 94.2\% $). We also visualize the learned ConvNet models, which qualitatively demonstrates the effectiveness of temporal segment network and the proposed good practices.

Compressed Video Action Recognition

Abstract: Training robust deep video representations has proven to be much more challenging than learning deep image representations. This is in part due to the enormous size of raw video streams and the high temporal redundancy; the true and interesting signal is often drowned in too much irrelevant data. Motivated by that the superfluous information can be reduced by up to two orders of magnitude by video compression (using H.264, HEVC, etc.), we propose to train a deep network directly on the compressed video. This representation has a higher information density, and we found the training to be easier. In addition, the signals in a compressed video provide free, albeit noisy, motion information. We propose novel techniques to use them effectively. Our approach is about 4.6 times faster than Res3D and 2.7 times faster than ResNet-152. On the task of action recognition, our approach outperforms all the other methods on the UCF-101, HMDB-51, and Charades dataset.

Compressed Video Action Recognition

Overview of state-of-the-art video analysis models. “Compressed” denotes the usage of compressed videos. “Mobile” denotes whether a model is applicable on mobile devices.

In this paper, we present a novel video analysis task, called mobile video action recognition, to fill in the above gap in video analysis (also see Table {{table:227eb76f-bf3b-4614-9a3c-5c742266a1a7}} ). Specifically, mobile video action recognition aims to perform efficient action recognition with compressed videos on mobile devices, considering the limit of storage and computation. This novel task meets the tendency of video analysis on mobile phones. On one hand, action recognition can help to automatically tag users' videos on social networking apps (e.g., Facebook and Instagram) before uploading from mobile phones. On the other hand, conducting video analysis on mobile devices can reduce the overload and computation on the cloud server. The key challenges of mobile video action recognition are: 1) how to design a lightweight and high-performance framework applicable on mobile devices; 2) how to extract meaningful representations from compressed videos. Thanks to the rapid development of neural network compression {{cite:bbbb659}}, {{cite:cc5b632}}, {{cite:ae4a5f0}}, {{cite:60a31d6}}, the compressed small networks can be used as the backbone for overcoming the first challenge, which balance the effectiveness and efficiency. In addition, compressed videos (e.g., MPEG-4) have been exploited {{cite:71a5a7c}} to replace raw videos for saving storage: multiple modalities (including RGB I-frame (I), low-resolution motion vector (MV) and residual (R) {{cite:b110805}}) are extracted to avoid bringing in extra optical flow data. However, {{cite:b110805}} used three deep models to process I-frames, motion vectors and residuals, and simply summed prediction scores from all these modalities in an ensemble manner. We argue that the ensemble strategy cannot fully capture the inner interactions between different modalities from compressed videos.

According to the above considerations, we propose a lightweight framework to solve the mobile video action recognition task. Specifically, we employ MobileNetV2 {{cite:cc5b632}} as the backbone network to process the multiple modalities (including RGB I-frame I, motion vectors MV, and residual R) extracted from compressed videos. Instead of the score ensemble strategy used in {{cite:b110805}}, we further propose a novel Temporal Trilinear Pooling (TTP) for fusing the extracted multiple modalities. In addition to motion vectors, we also put forward a temporal fusion method by combining successive key frames to achieve better temporal representation. As shown in Table {{table:227eb76f-bf3b-4614-9a3c-5c742266a1a7}} , the proposed TTP does fill in the gap in video analysis. Particularly, the efficiency test on a mobile device indicates that our TTP can perform mobile video action recognition at about 40FPS (see Table {{table:b4b7b40b-09cf-4a74-b6b0-3c31dd733b12}} ). Moreover, as shown in Figure {{figure:da8ed6a4-64d8-4f69-ab1e-5eabca5adf0f}} , our TTP achieves competitive performance but with extremely fewer parameters, as compared to state-of-the-art methods for video action recognition. This observation is further supported by the extensive results reported on two action recognition benchmarks (see Table {{table:786f1f1c-a994-4eb2-bd5b-6dcbaf196e4f}} ).