unrelated (75%), disagree (10%), agree (9%), discuss (6%)

comment (41%), unrelated (38%), support (13%), refute (8%)

argument against (56%), argument for (44%)

The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants

Abstract: Reasoning is a crucial part of natural language argumentation. To comprehend an argument, one must analyze its warrant, which explains why its claim follows from its premises. As arguments are highly contextualized, warrants are usually presupposed and left implicit. Thus, the comprehension does not only require language understanding and logic skills, but also depends on common sense. In this paper we develop a methodology for reconstructing warrants systematically. We operationalize it in a scalable crowdsourcing process, resulting in a freely licensed dataset with warrants for 2k authentic arguments from news comments. On this basis, we present a new challenging task, the argument reasoning comprehension task. Given an argument with a claim and a premise, the goal is to choose the correct implicit warrant from two options. Both warrants are plausible and lexically close, but lead to contradicting claims. A solution to this task will define a substantial step towards automatic warrant reconstruction. However, experiments with several neural attention and language models reveal that current approaches do not suffice.

Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims

Abstract: One key consequence of the information revolution is a significant increase and a contamination of our information supply. The practice of fact checking won't suffice to eliminate the biases in text data we observe, as the degree of factuality alone does not determine whether biases exist in the spectrum of opinions visible to us. To better understand controversial issues, one needs to view them from a diverse yet comprehensive set of perspectives. For example, there are many ways to respond to a claim such as "animals should have lawful rights", and these responses form a spectrum of perspectives, each with a stance relative to this claim and, ideally, with evidence supporting it. Inherently, this is a natural language understanding task, and we propose to address it as such. Specifically, we propose the task of substantiated perspective discovery where, given a claim, a system is expected to discover a diverse set of well-corroborated perspectives that take a stance with respect to the claim. Each perspective should be substantiated by evidence paragraphs which summarize pertinent results and facts. We construct PERSPECTRUM, a dataset of claims, perspectives and evidence, making use of online debate websites to create the initial data collection, and augmenting it using search engines in order to expand and diversify our dataset. We use crowd-sourcing to filter out noise and ensure high-quality data. Our dataset contains 1k claims, accompanied with pools of 10k and 8k perspective sentences and evidence paragraphs, respectively. We provide a thorough analysis of the dataset to highlight key underlying language understanding challenges, and show that human baselines across multiple subtasks far outperform ma-chine baselines built upon state-of-the-art NLP techniques. This poses a challenge and opportunity for the NLP community to address.

A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking

Abstract: Automated fact-checking based on machine learning is a promising approach to identify false information distributed on the web. In order to achieve satisfactory performance, machine learning methods require a large corpus with reliable annotations for the different tasks in the fact-checking process. Having analyzed existing fact-checking corpora, we found that none of them meets these criteria in full. They are either too small in size, do not provide detailed annotations, or are limited to a single domain. Motivated by this gap, we present a new substantially sized mixed-domain corpus with annotations of good quality for the core fact-checking tasks: document retrieval, evidence extraction, stance detection, and claim validation. To aid future corpus construction, we describe our methodology for corpus creation and annotation, and demonstrate that it results in substantial inter-annotator agreement. As baselines for future research, we perform experiments on our corpus with a number of model architectures that reach high performance in similar problem settings. Finally, to support the development of future models, we provide a detailed error analysis for each of the tasks. Our results show that the realistic, multi-domain setting defined by our data poses new challenges for the existing models, providing opportunities for considerable improvement by future systems.

Will-They-Won't-They: A Very Large Dataset for Stance Detection on Twitter

Abstract: We present a new challenging stance detection dataset, called Will-They-Won't-They (WT-WT), which contains 51,284 tweets in English, making it by far the largest available dataset of the type. All the annotations are carried out by experts; therefore, the dataset constitutes a high-quality and reliable benchmark for future research in stance detection. Our experiments with a wide range of recent state-of-the-art stance detection systems show that the dataset poses a strong challenge to existing models in this domain.

Cross-topic Argument Mining from Heterogeneous Sources Using Attention-based Neural Networks

Abstract: Argument mining is a core technology for automating argument search in large document collections. Despite its usefulness for this task, most current approaches to argument mining are designed for use only with specific text types and fall short when applied to heterogeneous texts. In this paper, we propose a new sentential annotation scheme that is reliably applicable by crowd workers to arbitrary Web texts. We source annotations for over 25,000 instances covering eight controversial topics. The results of cross-topic experiments show that our attention-based neural network generalizes best to unseen topics and outperforms vanilla BiLSTM models by 6% in accuracy and 11% in F-score.

Abstract: Stance detection is an important component of understanding hidden influences in everyday life. Since there are thousands of potential topics to take a stance on, most with little to no training data, we focus on zero-shot stance detection: classifying stance from no training examples. In this paper, we present a new dataset for zero-shot stance detection that captures a wider range of topics and lexical variation than in previous datasets. Additionally, we propose a new model for stance detection that implicitly captures relationships between topics using generalized topic representations and show that this model improves performance on a number of challenging linguistic phenomena.

List of the 16 stance datasets we use and their characteristics (sorted by source, then alphabetically).

The most crucial of these, which has not been investigated, currently preventing cross-domain studies, is that the label inventories differ between the different settings, as illustrated in Table {{table:1a98ffe1-22dd-42ab-8a94-612aba53bf5d}} . The possible labels include not only variants of agree, disagree, and unrelated, but also ones that are difficult to cross-map, such as discuss, and question.

In this section, we provide a brief overview of the 16 stance datasets included in our study, and we show their key characteristics in Table {{table:1a98ffe1-22dd-42ab-8a94-612aba53bf5d}} . More details are given in the appendix (Section REF ). We further motivate the source groupings used in our experiments and analysis (Section REF ).

As is readily apparent from Table {{table:1a98ffe1-22dd-42ab-8a94-612aba53bf5d}} , the datasets differ based on the nature of the target and the context, as well as the stance labels.

Defining source groups/domains is an important part of this study, as they allow for better understanding of the relationship between datasets, which we leverage through domain adaptive modelling (Section ). Moreover, we use them to outline phenomena in the results that similar datasets share (Section ). Table {{table:1a98ffe1-22dd-42ab-8a94-612aba53bf5d}}  shows these dataset groupings.

We would also like to note that the use of large-scale Transformers requires a lot of computations and the use of GPUs/TPUs for training, which contributes to global warming {{cite:aa67632}}. This is a bit less of an issue in our case, as we do not train such models from scratch; rather, we fine-tune them on relatively small datasets. Moreover, running on a CPU for inference, once the model is fine-tuned, is perfectly feasible, and CPUs contribute much less to global warming.

Ehud Aharoni, Anatoly Polnarov, Tamar Lavee, Daniel Hershcovich, Ran Levy, Ruty
Rinott, Dan Gutfreund, and Noam Slonim. 2014.
A benchmark dataset for
automatic detection of claims and evidence in the context of controversial
topics.
In Proceedings of the First Workshop on Argumentation Mining,
pages 64–68, Baltimore, Maryland. Association for Computational Linguistics.
Emily Allaway and Kathleen McKeown. 2020.
Zero-Shot Stance Detection: A Dataset and Model using
Generalized Topic Representations.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 8913–8931, Online. Association
for Computational Linguistics.
Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper
Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019.
MultiFC: A
real-world multi-domain dataset for evidence-based fact checking of claims.
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages 4685–4697, Hong Kong,
China. Association for Computational Linguistics.
Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina
Bontcheva. 2016.
Stance detection with
bidirectional conditional encoding.
In Proceedings of the 2016 Conference on Empirical Methods in
Natural Language Processing, pages 876–885, Austin, Texas. Association for
Computational Linguistics.
Isabelle Augenstein, Sebastian Ruder, and Anders Søgaard. 2018.
Multi-task learning of
pairwise sequence classification tasks over disparate label spaces.
In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long Papers), pages 1896–1906, New Orleans,
Louisiana. Association for Computational Linguistics.
Roy Bar-Haim, Indrajit Bhattacharya, Francesco Dinuzzo, Amrita Saha, and Noam
Slonim. 2017.
Stance
classification of context-dependent claims.
In Proceedings of the 15th Conference of the European Chapter
of the Association for Computational Linguistics: Volume 1, Long Papers,
pages 251–261, Valencia, Spain. Association for Computational Linguistics.
Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo
Neves. 2020.
TweetEval: Unified benchmark and comparative evaluation for tweet
classification.
In Findings of the Association for Computational Linguistics:
EMNLP 2020, pages 1644–1650, Online. Association for Computational
Linguistics.
Genady Beryozkin, Yoel Drori, Oren Gilon, Tzvika Hartman, and Idan Szpektor.
2019.
A joint named-entity
recognizer for heterogeneous tag-sets using a tag hierarchy.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 140–150, Florence, Italy. Association
for Computational Linguistics.
Johannes Bjerva, Wouter Kouw, and Isabelle Augenstein. 2019.
Back to the Future – Sequential Alignment of Text Representations.
In In Proceedings of the 34th AAAI Conference on Artificial
Intelligence.
John Blitzer, Ryan McDonald, and Fernando Pereira. 2006.
Domain adaptation
with structural correspondence learning.
In Proceedings of the 2006 Conference on Empirical Methods in
Natural Language Processing, pages 120–128, Sydney, Australia. Association
for Computational Linguistics.
Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and Jiwei Li. 2020.
Description based text classification with reinforcement learning.
arXiv preprint arXiv:2002.03067.
Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit S Dhillon.
2020.
Taming pretrained transformers for extreme multi-label text
classification.
In Proceedings of the 26th ACM SIGKDD International Conference
on Knowledge Discovery & Data Mining, pages 3163–3171.
Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth.
2019.
Seeing things from a
different angle:discovering diverse perspectives about claims.
In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers), pages 542–557, Minneapolis,
Minnesota. Association for Computational Linguistics.
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou,
Flavio Toxvaerd, and Nigel Collier. 2020.
Will-they-won't-they: A very large dataset for stance detection on
Twitter.
In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 1715–1724, Online. Association for
Computational Linguistics.
Hal Daumé III. 2007.
Frustratingly easy
domain adaptation.
In Proceedings of the 45th Annual Meeting of the Association of
Computational Linguistics, pages 256–263, Prague, Czech Republic.
Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
BERT: Pre-training of deep bidirectional Transformers for
language understanding.
In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT '19, pages 4171–4186, Minneapolis, Minnesota, USA.
William Ferreira and Andreas Vlachos. 2016.
Emergent: a novel
data-set for stance classification.
In Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, pages 1163–1168, San Diego, California. Association for
Computational Linguistics.
Yaroslav Ganin and Victor Lempitsky. 2015.
Unsupervised domain adaptation by backpropagation.
In Proceedings of the 32nd International Conference on Machine
Learning, volume 37 of Proceedings of Machine Learning Research,
pages 1180–1189, Lille, France.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo
Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.
2016.
Domain-adversarial training of neural networks.
J. Mach. Learn. Res., 17(1):2096–2030.
Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkaitz Zubiaga,
Kalina Bontcheva, and Leon Derczynski. 2019.
SemEval-2019 task
7: RumourEval, determining rumour veracity and support for rumours.
In Proceedings of the 13th International Workshop on Semantic
Evaluation, pages 845–854, Minneapolis, Minnesota, USA. Association for
Computational Linguistics.
Tao Gui, Qi Zhang, Haoran Huang, Minlong Peng, and Xuanjing Huang. 2017.
Part-of-speech tagging
for Twitter with adversarial neural networks.
In Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing, pages 2411–2420, Copenhagen, Denmark.
Association for Computational Linguistics.
Jiang Guo, Darsh Shah, and Regina Barzilay. 2018.
Multi-source domain
adaptation with mixture of experts.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 4694–4703, Brussels, Belgium.
Association for Computational Linguistics.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy,
Doug Downey, and Noah A. Smith. 2020.
Don't stop
pretraining: Adapt language models to domains and tasks.
In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 8342–8360, Online. Association for
Computational Linguistics.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018.
The argument reasoning
comprehension task: Identification and reconstruction of implicit warrants.
In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long Papers), pages 1930–1940, New Orleans,
Louisiana. Association for Computational Linguistics.
Xiaochuang Han and Jacob Eisenstein. 2019.
Unsupervised
domain adaptation of contextualized embeddings for sequence labeling.
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages 4238–4248, Hong Kong,
China. Association for Computational Linguistics.
Andreas Hanselowski, Avinesh PVS, Benjamin Schiller, Felix Caspelherr, Debanjan
Chaudhuri, Christian M. Meyer, and Iryna Gurevych. 2018.
A retrospective
analysis of the fake news challenge stance-detection task.
In Proceedings of the 27th International Conference on
Computational Linguistics, pages 1859–1874, Santa Fe, New Mexico, USA.
Association for Computational Linguistics.
Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna
Gurevych. 2019.
A richly annotated
corpus for different tasks in automated fact-checking.
In Proceedings of the 23rd Conference on Computational Natural
Language Learning (CoNLL), pages 493–503, Hong Kong, China. Association for
Computational Linguistics.
Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2021.
A survey on stance detection for mis-and disinformation
identification.
arXiv preprint arXiv:2103.00242.
K. Hasan and Vincent Ng. 2014.
Why are you taking this stance? identifying and classifying reasons
in ideological debates.
In EMNLP.
Kazi Saidul Hasan and Vincent Ng. 2013.
Stance
classification of ideological debates: Data, models, features, and
constraints.
In Proceedings of the Sixth International Joint Conference on
Natural Language Processing, pages 1348–1356, Nagoya, Japan. Asian
Federation of Natural Language Processing.
Tamanna Hossain, Robert L. Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean
Young, and Sameer Singh. 2020.
COVIDLies: Detecting COVID-19 misinformation on social media.
In Proceedings of the 1st Workshop on NLP for COVID-19
(Part 2) at EMNLP 2020, Online. Association for Computational Linguistics.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017.
Bag of tricks for
efficient text classification.
In Proceedings of the 15th Conference of the European Chapter
of the Association for Computational Linguistics: Volume 2, Short Papers,
pages 427–431, Valencia, Spain. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015.
Adam: A method for
stochastic optimization.
In 3rd International Conference on Learning Representations,
ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
Proceedings.
Dilek Küçük and Fazli Can. 2020.
Stance detection: A survey.
ACM Comput. Surv., 53(1).
Taku Kudo and John Richardson. 2018.
SentencePiece: A
simple and language independent subword tokenizer and detokenizer for neural
text processing.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations, pages 66–71, Brussels,
Belgium. Association for Computational Linguistics.
Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018.
What's in a domain?
learning domain-robust text representations using adversarial training.
In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 2 (Short Papers), pages 474–479, New Orleans,
Louisiana. Association for Computational Linguistics.
Bill Yuchen Lin and Wei Lu. 2018.
Neural adaptation
layers for cross-domain named entity recognition.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 2012–2022, Brussels, Belgium.
Association for Computational Linguistics.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. 2018.
Detecting and correcting for label shift with black box predictors.
In International conference on machine learning, pages
3122–3130. PMLR.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
RoBERTa: A robustly optimized BERT pretraining approach.
ArXiv, abs/1907.11692.
Edward Loper and Steven Bird. 2002.
Nltk: The natural
language toolkit.
In Proceedings of the ACL-02 Workshop on Effective Tools and
Methodologies for Teaching Natural Language Processing and Computational
Linguistics - Volume 1, ETMTNLP '02, page 63–70, USA. Association for
Computational Linguistics.
Xiaofei Ma, Peng Xu, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2019.
Domain adaptation with
BERT-based domain classification and data selection.
In Proceedings of the 2nd Workshop on Deep Learning Approaches
for Low-Resource NLP (DeepLo 2019), pages 76–83, Hong Kong, China.
Association for Computational Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008.
Visualizing
data using t-sne.
Journal of Machine Learning Research, 9(86):2579–2605.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin
Cherry. 2016.
SemEval-2016 task
6: Detecting stance in tweets.
In Proceedings of the 10th International Workshop on Semantic
Evaluation (SemEval-2016), pages 31–41, San Diego, California.
Association for Computational Linguistics.
Saif M. Mohammad, Parinaz Sobhani, and Svetlana Kiritchenko. 2017.
Stance and sentiment in
tweets.
ACM Trans. Internet Technol., 17(3).
Nikolaos Pappas and James Henderson. 2019.
Gile: A generalized input-label embedding for text classification.
Transactions of the Association for Computational Linguistics,
7:139–155.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.
GloVe: Global
vectors for word representation.
In Proceedings of the 2014 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar.
Association for Computational Linguistics.
Dean Pomerleau and Delip Rao. 2017.
Fake news challenge stage
1 (FNC-I): Stance detection.
Vahed Qazvinian, Emily Rosengren, Dragomir R. Radev, and Qiaozhu Mei. 2011.
Rumor has it:
Identifying misinformation in microblogs.
In Proceedings of the 2011 Conference on Empirical Methods in
Natural Language Processing, pages 1589–1599, Edinburgh, Scotland, UK.
Association for Computational Linguistics.
Nils Rethmeier and Isabelle Augenstein. 2020.
Long-Tail Zero and Few-Shot Learning via Contrastive Pretraining on
and for Small Data.
arXiv preprint arXiv:2010.01061.
Alexander Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. 2020.
Adapt or
get left behind: Domain adaptation through BERT language model finetuning
for aspect-target sentiment classification.
In Proceedings of the 12th Language Resources and Evaluation
Conference, pages 4933–4941, Marseille, France. European Language Resources
Association.
Benjamin Schiller, Johannes Daxenberger, and Iryna Gurevych. 2020.
Stance detection benchmark: How robust is your stance detection?
arXiv preprint arXiv:2001.01565.
Parinaz Sobhani, Diana Inkpen, and S. Matwin. 2015.
From argumentation mining to stance classification.
In ArgMining@HLT-NAACL.
Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017.
A dataset for
multi-target stance detection.
In Proceedings of the 15th Conference of the European Chapter
of the Association for Computational Linguistics: Volume 2, Short Papers,
pages 551–557, Valencia, Spain. Association for Computational Linguistics.
Swapna Somasundaran and Janyce Wiebe. 2009.
Recognizing
stances in online debates.
In Proceedings of the Joint Conference of the 47th Annual
Meeting of the ACL and the 4th International Joint Conference on Natural
Language Processing of the AFNLP, pages 226–234, Suntec, Singapore.
Association for Computational Linguistics.
Swapna Somasundaran and Janyce Wiebe. 2010.
Recognizing
stances in ideological on-line debates.
In Proceedings of the NAACL HLT 2010 Workshop on
Computational Approaches to Analysis and Generation of Emotion in Text,
pages 116–124, Los Angeles, CA. Association for Computational Linguistics.
Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna
Gurevych. 2018.
Cross-topic argument
mining from heterogeneous sources.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 3664–3674, Brussels, Belgium.
Association for Computational Linguistics.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.
Energy and policy
considerations for deep learning in NLP.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, ACL '19, pages 3645–3650, Florence, Italy.
Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014.
Learning
sentiment-specific word embedding for Twitter sentiment classification.
In Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 1555–1565,
Baltimore, Maryland. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.
Attention is all you need.
In Proceedings of the Annual Conference on Neural Information
Processing Systems, NIPS '17, pages 5998–6008, Long Beach, CA, USA.
Marilyn Walker, Jean Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012.
A corpus
for research on deliberation and debate.
In Proceedings of the Eighth International Conference on
Language Resources and Evaluation (LREC'12), pages 812–817, Istanbul,
Turkey. European Language Resources Association (ELRA).
Rui Wang, Deyu Zhou, Mingmin Jiang, Jiasheng Si, and Y. Yang. 2019.
A survey on opinion mining: From stance to product aspect.
IEEE Access, 7:41101–41124.
Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021.
Disembodied machine learning: On the illusion of objectivity in nlp.
arXiv preprint arXiv:2101.11974.
Dustin Wright and Isabelle Augenstein. 2020.
Transformer
based multi-source domain adaptation.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 7963–7974, Online. Association
for Computational Linguistics.
Ting Yao, Yingwei Pan, Chong-Wah Ngo, Houqiang Li, and Tao Mei. 2015.
Semi-supervised domain adaptation with subspace learning for visual
recognition.
In Proceedings of the IEEE conference on Computer Vision and
Pattern Recognition, pages 2142–2150.
Arkaitz Zubiaga. 2018.
A longitudinal assessment of the persistence of twitter datasets.
Journal of the Association for Information Science and
Technology, 69(8):974–984.
Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter.
2018.
Detection and resolution of
rumours in social media: A survey.
ACM Comput. Surv., 51(2).

Fine-Tuning and Hyper-Parameters
[nosep,leftmargin=*]
All models use Adam {{cite:402d5e1}} with weight decay 1e-8, {{formula:96ba20ec-9786-4272-894d-6ed116ff3d2c}}  0.9, {{formula:bf1382de-630c-4560-a4d9-bd9fd4fdf036}}  0.999), {{formula:8748fd03-2212-49b9-87e5-3d2ca90b7f39}}  1e-08, and are trained for 5 epochs with batch size 64, and maximum sequence length of 100 tokens.

RoBERTa {{cite:20a2fee}} is trained w/ LR 1e-05, warmup 0.06, BERT {{cite:4c6a06f}} is trained w/ LR 3e-05, warmup 0.1.

The aforementioned hyper-parameters are selected on the dev set.

We also choose the best model checkpoint based on the performance on the dev set.

For the MTL/MoE models, we sample each batch from a single randomly selected dataset/domain.

We used the same seed for all experiments.

Each experiment took around 1h 15m on a single NVIDIA V100 GPU using half precision.

Dataset Analysis
Datasets
Example from each dataset with stance target. The texts with ... are shortened due to space limitations.
lp3.5cmXl
Dataset  Target  Context  Labelarc  States do not need special schools for the deaf  In the early 90's I was studying Maths of Finance at university and fees and charges were just starting to raise their ugly heads. In hindsight Australia is about 20 years ahead of the US with ... unrelatedargmin  cloning  In Humanity Enhanced , I challenge the idea that children conceived through SCNT would have their autonomy violated – or would somehow lack or lose autonomy – in any sense inapplicable to “ ordinary ” children .  argument_foremergent  Jess Smith of Chatham, Kent was the smiling sun baby in the Teletubbies TV show  Canterbury Christ Church University student Jess Smith, from Chatham, starred as Teletubbies sun  forfnc1  Nigeria announces truce with Boko Haram; fate of schoolgirls unclear  Well, here’s the creepiest thing you’ll read all day. Australian Dylan Thomas found a tropical spider burrowed UNDER his skin after returning from a trip to Bali. ... unrelatediac1  marijuana legalization  [P1] Instead of rewriting what I have already written numerous times, I will post a copy of a recent post I made on another thread regarding this same topic. You can find the original post here.[P2] I enjoy the experience of getting drunk, high, or anything else like that...... proibmcs  This house believes atheism is the only way  God is improbable  promtsd  Hilary Clinton  Love that #democratic primary is talking bout real issues #BernieSanders made a great case now let's hear from #HillaryClinton #DemTownHall  noneperspectrum  The lack of investment in teachers is the greatest barrier to achieving universal primary education  It should be social policy to make teaching careers more desirable  supportpoldeb  should the us have universal healthcare  Yes, my posts all turned up backward because I replied in the order I read.Props for all the research you did. I still wholeheartedly disagree.Here's one article I came across that summarizes my entire point of view. http://url In the end, perhaps we agree to disagree ... againstrumor  Sarah Palin getting divorced?  OneRiot.com - Palin Denies First Dude Divorce Rumors http://url  denyscd  –  First off, the only people that want to legalize pot are the liberals that sit around all day, living off wellfare and smoke drugs. They do this because they are to uneducated to get a job, and figure the liberal ... againstsemeval2016t6  Legalization of Abortion  @mrsdrjim did you know #wrp @BrianJeanWRP tried to get personhood going via federal #motion312. #SemST  nonesemeval2019t7  –  Wow, that is fascinating! I hope you never mock our proud Scandi heritage again.  commentsnopes  president-elect donald trump's inauguration will be the first presidential inauguration that rep. john lewis has skipped.  wrong (or lie)!  refutevast  public education  Public schools are the entire country's investment in an educated populace. They are our investment in a responsible, civil society. Everyone benefits when every citizen is able to read, write, understand history ... prowtwt  Anthem acquires Cigna  - #tuu i #yoo - Anthem Reaffirms Commitment to Its $47-Billion Bid for Cigna: Anthem stands by its $47-bill... http://url  support
{{table:ccfba332-36d9-43b1-845d-d39ee292cf50}}arc The Argument Reasoning Comprehension dataset has posts from the New York Times debate section on immigration and international affairs.
argmin The Argument Mining corpus presents arguments relevant to a particular topic from heterogenous texts. Topics include controversial keywords like death penalty and gun control.
emergent The Emergenthttp://www.emergent.info/ dataset is a collection of articles from rumour sites annotated by journalists.
fnc1 The Fake News Challenge dataset consists of news articles whose stance towards headlines is provided. It spans 300 topics from Emergent[5].
iac1 The Internet Argument Corpus V1 consists of Quote-Response pairs from a debating forum on topics related to US politics.
ibmcs This dataset expands the IBM argumentative structure dataset {{cite:bf8edfb}} to 55 topics and provides topic-claim pairs (from IBM Project Debaterhttp://www.research.ibm.com/artificial-intelligence/project-debater/) along with their stance annotations.
mtsd The Multi-Target Stance Detection dataset includes tweets related to the 2016 US Presidential election with a specific focus on multiple targets of interest expressed in each tweet.
perspectrum The Perspectrum dataset provides several perspectives towards a given claim gathered from a number of debating websites.
poldeb The Ideological On-Line Debates corpus provides opinion-target pairs from several debating platforms encapsulating different domains.
rumor The Rumor Has It dataset presents tweets for the task of Belief Classification, where users believe, question, or refute curated rumours.
vast The Varied Stance Topics dataset consists of topic–comment pairs from the The New York Times Room for Debate section. The dataset covers a large variety of topics in order to facilitate zero-shot learning on new unseen topics.
wtwt Will-They-Won't-They presents a large number of annotated tweets from the financial domain relating to five merger and acquisition operations.
scd The Stance Classification dataset provides debate posts from four domains including Obama and Gay Rights. As highlighted in Table {{table:8f04ea7a-9666-49a2-b755-ac434880ca51}} , while the posts are gathered from defined domains, they are not part of the training set and need to be inferred.
semeval2016t6 The SemEval-2016 Task 6 dataset provides tweet–target pairs for 5 targets including Atheism, Feminist Movement, and Climate Change.
semeval2019t7 The SemEval=2019 Task 9 dataset aims to model authors' stance towards a particular rumour. It provides annotated tweets supporting, denying, querying, or commenting on the rumour.
snopes The Snopes dataset provides several controversial claims and their corresponding evidence texts from the US based fact-checking website Snopeshttp://www.snopes.com annotated for the text in support of, refuting or having no stance towards a claim.
Examples for each dataset are shown in Table {{table:ccfba332-36d9-43b1-845d-d39ee292cf50}} .

Data Splits
We could not reconstruct some of the Social Media datasets in full (marked with a * symbol in Table {{table:8f04ea7a-9666-49a2-b755-ac434880ca51}} ), as with only tweet IDs, we could not obtain the actual tweet text in some cases. This is a known phenomenon in Twitter: with time, older tweets become unavailable for various reasons, such as tweets/accounts being deleted or accounts being made private {{cite:779ee18}}. The missing tweets are evenly distributed among the splits of the datasets except for rumor, where we chose a topic for the test set for which all example texts were available.
Here, we provide details about the splits we use for the datasets in cases where there is a deviation from the original. For the datasets in common, we use the splitting done by {{cite:4a649a5}}. We further tried to enforce a larger domain diversity between the train, dev, and test sets; hereby, we put (whenever possible) the examples from a particular topic (domain) strictly into a single split.
argmin All non-arguments were removed. The train, develop and test data splits consist of five, one, and two topics, respectively.
iac1 Split with no intersection of topics between train, development, and test set.
ibmcs Pre-defined train and test split. Further, 10% of the training data was reserved for the dev set.
mtsd We use the pre-defined splits, but we create two pairs for each example. A positive and a negative one towards a target.
poledb We use the domains Healthcare, Guns, Gay Rights and God for training, Abortion for development, and Creation for testing.
rumor We use the airfrance rumour for our test set, and we split the remaining data in ratio 9:1 for training and development, respectively.
wtwt We use DIS_FOXA operation for testing, AET_HUM for development, and the rest for training. To standardize the targets, we re-write them as sentences, i.e., company X acquires company Y.
scd Split with Marijuana for development and Obama test, and the rest for training.
semeval2016t6 Split to increase the size of the development set.
snopes Splits adjusted for compatibility with the stance setup. Rumours and their evidence were extracted and converted into target–context pairs.

Overlap Statistics
Percentage of overlap between dev/test and the training dataset. The (T)arget and the (C)ontext columns show the overlap in the respective individual fields, while (F)ull shows the overall overlap.
1.00!
Dev
Test
% of split in Train
F
T
C
F
T
C
arc
1.9
100.0
93.7
1.5
100.0
93.8
iac1
0.0
0.0
0.2
0.0
0.0
0.1
perspectrum
1.5
1.5
37.2
0.0
0.0
26.4
poldeb
0.0
0.0
0.0
0.0
0.0
0.0
scd
—
—
0.0
—
—
0.0
emergent
0.0
0.0
3.0
0.0
0.0
1.7
fnc1
1.7
100.0
99.8
0.0
0.6
0.9
snopes
0.0
0.0
0.0
0.0
0.0
0.3
mtsd
0.8
100.0
1.5
0.3
100.0
0.5
rumor
17.6
100.0
17.6
0.0
0.0
0.0
semeval2016t6
0.0
100.0
0.0
0.0
100.0
0.0
semeval2019t7
—
—
1.4
—
—
4.8
wtwt
0.0
0.0
11.4
0.0
0.0
0.0
argmin
0.0
0.0
0.0
0.0
0.0
0.0
ibmcs
0.0
100.0
1.0
0.0
0.0
0.2
vast
0.0
43.6
0.0
0.0
49.5
0.0
{{table:1ec9d978-9a10-4638-a9f1-f97a8350f942}}{{table:55c8826a-693e-4b22-ba29-f4478cfd4789}}Next, in Table {{table:55c8826a-693e-4b22-ba29-f4478cfd4789}} , we examine the proportion of contexts and targets from the dev and the test datasets that are also present in the training split. We do not change the original data in any way, and we use the splits as described in Section REF .
Statistics about the sub-word tokens for each dataset (using RoBERTaBase's tokenizer). Numbers in parenthesis show word count using NLTK tokenizer.
1.00!Tokenization
Words
Tokens (Words)
Tokens

{{formula:52eb3e2d-d45a-4203-bccc-4028c47e4cfd}} Unique{{formula:c2561768-ee3b-44bb-937d-ca567fa41837}}
Mean
25%
Median
Max
arc
27,835
126.0 (118.3)
84
116
286
argmin
17,990
30.6 (28.6)
20
28
208
emergent
6,940
27.6 (23.1)
22
26
111
fnc1
40,738
503.2 (432.2)
279
413
6,182
iac1
88,478
1,554.7 (1,347.9)
132
390
104,034
ibmcs
5,007
23.4 (21.9)
18
23
55
mtsd
9,799
36.3 (25.9)
33
36
65
perspectrum
9,999
22.4 (20.4)
17
21
75
poldeb
40,422
178.0 (160.6)
55
112
2,144
rumor
15,801
38.4 (24.4)
32
39
78
scd
23,592
151.4 (134.8)
39
78
6,358
semeval2016t6
15,016
32.8 (22.1)
28
33
68
semeval2019t7
20,789
33.5 (22.0)
17
27
1,466
snopes
33,896
53.4 (45.5)
40
51
327
vast
24,644
123.2 (115.7)
80
114
271
wtwt
102,672
45.7 (23.1)
39
46
193
{{table:84ea31ca-c648-42b2-8d98-b42b77d48f0b}}{{table:efbee143-f97f-411b-8bc0-f28382e21b78}}We further show statistics about the datasets in terms of the number of words and sub-words (see Table {{table:efbee143-f97f-411b-8bc0-f28382e21b78}} ). The first column is the number of unique words in each dataset from NLTK's casual tokenizer {{cite:e9dccc6}}, which retains the casing of words. We observe that the datasets with the largest vocabularies are (i) those with higher numbers of examples (fnc1 and wtwt), (ii) whose contexts are threads rather than single posts (iac1 has over 1,300 words on average), and (iii) that cover diverse topics such as
poldeb with six unrelated ones. In contrast, small or narrow datasets such as ibmcs have the smallest vocabularies (fewer than 5,000 words). In the subsequent columns, we report statistics in terms of sub-words (i.e., SentencePieces {{cite:ca2c5dd}} from RoBERTaBase's tokenizer). With that, we want to present the expected coverage in terms of tokens for a pre-trained model. On average, most of the datasets are well under 100 tokens in length, which is commonly observed for tweets,Tweets have a strict character limit. Depending on the time period, the limit can vary. but some datasets have a higher average number of tokens, e.g., debate-related datasets such as arc, poldeb, scd, vast fit in 200 tokens on average, which is also the case for datasets containing large news articles or that use social media threads as context (fnc1, iac1), where the average length is over 500.
Proportion of word types in the dataset in row {{formula:88b82fe9-a941-4f6a-977d-96713d32ad1c}}  that are also present in the dataset in column {{formula:52034879-265d-4586-a091-08fd0850e543}}  (stop-words removed, and case preserved).
{{figure:d8144871-355a-4036-8ef5-5b3c97a6e54f}}Embeddings spaces of the label names' representations (PCA) from different embedding models.
[FastText ]{{figure:313932e9-3ea8-4066-96f6-5bb2ef1fc0d7}}{{figure:8dd3c610-0346-4b6d-9c2f-cf4ff68502e1}}{{figure:f12707f5-9226-4b39-9677-4da189dcb370}}[RoBERTa-sentiment ]{{figure:95af5487-d154-47e9-b04d-a86b10a21f50}}[SSWE ]{{figure:12619dcf-1de9-4d7f-9a9a-f7ae39aed576}}{{figure:b4e56f2d-c99f-4a9e-b2f5-ad6f84f6dc38}}Finally, Figure {{figure:d8144871-355a-4036-8ef5-5b3c97a6e54f}}  shows the proportion of word overlaps between datasets. The numbers in each cell shows how much of the word types in dataset {{formula:7573f3c1-97c7-4839-b71d-729f470a20f4}}  (row) are contained in the dataset {{formula:8de1c547-65dc-45e7-b685-99c0d1507852}}  (column). For example, if we take the first column in the last row (vast {{formula:13b33ee9-17e1-425e-b3d2-f527eb7fee34}}  arc), we see that 97% of the words in vast are also present in arc. Similarly, if we look at the first row and the last column, we can see that 86% of the words in arc are also in vast. For convenience, we sort the columns and the rows by their sources (see Table {{table:1a98ffe1-22dd-42ab-8a94-612aba53bf5d}} ). We can see that the datasets with the largest vocabularies (iac1 and wtwt) have low overlap with the other datasets, including with each other, going up to only {{formula:1bb826f0-673d-46c7-b08a-f74f4f4288f8}}  (row-wise). In contrast, when looking at how many words in other datasets are contained in them (column-wise), we see that iac1 has at least 50% vocabulary overlap with the other datasets, even with ones from different sources. While wtwt's overlaps are moderate between {{formula:689beda7-9121-4105-be45-6f9eec432a06}} , but not as prominent as iac1, the latter is expected due to the texts being from social media, and covering a very narrow topic such as company acquisitions. For datasets that are either small in size or have few topics (emergant, ibmcs, perspectrum), we see that moderate to large part of their vocabularies are contained in other datasets; yet, the opposite in not true. Furthermore, the social media datasets are orthogonal to each other, as is apparent in their cross-overlaps (both row-wise and column-wise): reaching at most 50%, except for wtwt. That is also observed in measuring how much of other datasets' vocabulary they contain (column-wise for social media).

Embedding Label Spaces
In this section, we present the embeddings based on the labels' names. We explore five sets of embeddings: well-established – fastText {{cite:5ae34b1}}, GloVe {{cite:d8b32ef}}, RoBERTa {{cite:20a2fee}}, and sentiment-enriched – Sentiment-specific word embedding (SSWE) {{cite:5497d54}}, RoBERTa Twitter Sentiment (roberta-sentiment) {{cite:7d96e62}}. Our motivation for including the latter is that the names of the stance labels are often sentiment-related, and thus encoding that information into the latent space could yield better grouping. We encode each label with the corresponding word from the embedding's directory for non-contextualized embeddings, if the name contains multiple words, e.g., argument for, then we split on white-space and take the average of each word's embeddings. For RoBERTa-based models, we take the representation from the [CLS] token. Finally, we project the obtained vectors into two dimensions using PCA, the resulting plots are shown in Figure {{figure:b4e56f2d-c99f-4a9e-b2f5-ad6f84f6dc38}} .


Explainable Automated Fact-Checking for Public Health Claims

Abstract: Fact-checking is the task of verifying the veracity of claims by assessing their assertions against credible evidence. The vast majority of fact-checking studies focus exclusively on political claims. Very little research explores fact-checking for other topics, specifically subject matters for which expertise is required. We present the first study of explainable fact-checking for claims which require specific expertise. For our case study we choose the setting of public health. To support this case study we construct a new dataset PUBHEALTH of 11.8K claims accompanied by journalist crafted, gold standard explanations (i.e., judgments) to support the fact-check labels for claims. We explore two tasks: veracity prediction and explanation generation. We also define and evaluate, with humans and computationally, three coherence properties of explanation quality. Our results indicate that, by training on in-domain data, gains can be made in explainable, automated fact-checking for claims which require specific expertise.

Crowdsourcing Multiple Choice Science Questions

Abstract: We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-specific text and a small set of existing questions. It produces model suggestions for document selection and answer distractor choice which aid the human question generation process. With this method we have assembled SciQ, a dataset of 13.7K multiple choice science exam questions. We demonstrate that the method produces in-domain questions by providing an analysis of this new dataset and by showing that humans cannot distinguish the crowdsourced questions from original questions. When using SciQ as additional training data to existing questions, we observe accuracy improvements on real science exams.

Evidence Inference 2.0: More Data, Better Models

Abstract: How do we most effectively treat a disease or condition? Ideally, we could consult a database of evidence gleaned from clinical trials to answer such questions. Unfortunately, no such database exists; clinical trial results are instead disseminated primarily via lengthy natural language articles. Perusing all such articles would be prohibitively time-consuming for healthcare practitioners; they instead tend to depend on manually compiled systematic reviews of medical literature to inform care. NLP may speed this process up, and eventually facilitate immediate consult of published evidence. The Evidence Inference dataset was recently released to facilitate research toward this end. This task entails inferring the comparative performance of two treatments, with respect to a given outcome, from a particular article (describing a clinical trial) and identifying supporting evidence. For instance: Does this article report that chemotherapy performed better than surgery for five-year survival rates of operable cancers? In this paper, we collect additional annotations to expand the Evidence Inference dataset by 25%, provide stronger baseline models, systematically inspect the errors that these make, and probe dataset quality. We also release an abstract only (as opposed to full-texts) version of the task for rapid model prototyping. The updated corpus, documentation, and code for new baselines and evaluations are available at http://evidence-inference.ebm-nlp.com/.

BioCreative V CDR task corpus: a resource for chemical disease relation extraction

Abstract: Community-run, formal evaluations and manually annotated text corpora are critically important for advancing biomedical text-mining research. Recently in BioCreative V, a new challenge was organized for the tasks of disease named entity recognition (DNER) and chemical-induced disease (CID) relation extraction. Given the nature of both tasks, a test collection is required to contain both disease/chemical annotations and relation annotations in the same set of articles. Despite previous efforts in biomedical corpus construction, none was found to be sufficient for the task. Thus, we developed our own corpus called BC5CDR during the challenge by inviting a team of Medical Subject Headings (MeSH) indexers for disease/chemical entity annotation and Comparative Toxicogenomics Database (CTD) curators for CID relation annotation. To ensure high annotation quality and productivity, detailed annotation guidelines and automatic annotation tools were provided. The resulting BC5CDR corpus consists of 1500 PubMed articles with 4409 annotated chemicals, 5818 diseases and 3116 chemical-disease interactions. Each entity annotation includes both the mention text spans and normalized concept identifiers, using MeSH as the controlled vocabulary. To ensure accuracy, the entities were first captured independently by two annotators followed by a consensus annotation: The average inter-annotator agreement (IAA) scores were 87.49% and 96.05% for the disease and chemicals, respectively, in the test set according to the Jaccard similarity coefficient. Our corpus was successfully used for the BioCreative V challenge tasks and should serve as a valuable resource for the text-mining research community. Database URL: http://www.biocreative.org/tasks/biocreative-v/track-3-cdr/

List of evaluation tasks and their metrics.

We conducted an evaluation of the models' capabilities by covering the 13 different tasks presented in Table {{table:e48ba6f9-4fa5-4fce-bfca-3337177a4dfd}} . The tasks were chosen to allow for a broad assessment in both clinical and biomedical domains, including tasks oriented for generative and classical model evaluation.


Quadratic video interpolation

Abstract: Video interpolation is an important problem in computer vision, which helps overcome the temporal limitation of camera sensors. Existing video interpolation methods usually assume uniform motion between consecutive frames and use linear models for interpolation, which cannot well approximate the complex motion in the real world. To address these issues, we propose a quadratic video interpolation method which exploits the acceleration information in videos. This method allows prediction with curvilinear trajectory and variable velocity, and generates more accurate interpolation results. For high-quality frame synthesis, we develop a flow reversal layer to estimate flow fields starting from the unknown target frame to the source frame. In addition, we present techniques for flow refinement. Extensive experiments demonstrate that our approach performs favorably against the existing linear models on a wide variety of video datasets.

FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation

Abstract: Most modern frame interpolation approaches rely on explicit bidirectional optical flows between adjacent frames, thus are sensitive to the accuracy of underlying flow estimation in handling occlusions while additionally introducing computational bottlenecks unsuitable for efficient deployment. In this work, we propose a flow-free approach that is completely end-to-end trainable for multi-frame video interpolation. Our method, FLAVR, leverages 3D spatio-temporal kernels to directly learn motion properties from unlabeled videos and greatly simplifies the process of training, testing and deploying frame interpolation models. As a result, FLAVR delivers up to 6× speed up compared to the current state-of-the-art methods for multi-frame interpolation while consistently demonstrating superior qualitative and quantitative results compared with prior methods on popular benchmarks including Vimeo-90K, Adobe-240FPS, and GoPro. Finally, we show that frame interpolation is a competitive self-supervised pre-training task for videos via demonstrating various novel applications of FLAVR including action recognition, optical flow estimation, and video object tracking. Code and trained models are provided in the supplementary material.

CDFI: Compression-Driven Network Design for Frame Interpolation

Abstract: DNN-based frame interpolation—that generates the intermediate frames given two consecutive frames—typically relies on heavy model architectures with a huge number of features, preventing them from being deployed on systems with limited resources, e.g., mobile devices. We propose a compression-driven network design for frame interpolation (CDFI), that leverages model pruning through sparsity-inducing optimization to significantly reduce the model size while achieving superior performance. Concretely, we first compress the recently proposed AdaCoF model and show that a 10× compressed AdaCoF performs similarly as its original counterpart; then we further improve this compressed model by introducing a multi-resolution warping module, which boosts visual consistencies with multi-level details. As a consequence, we achieve a significant performance gain with only a quarter in size compared with the original AdaCoF. Moreover, our model performs favorably against other state-of-the-arts in a broad range of datasets. Finally, the proposed compression-driven framework is generic and can be easily transferred to other DNN-based frame interpolation algorithm. Our source code is available at https://github.com/tding1/CDFI.

XVFI: eXtreme Video Frame Interpolation

Abstract: In this paper, we firstly present a dataset (X4K1000FPS) of 4K videos of 1000 fps with the extreme motion to the research community for video frame interpolation (VFI), and propose an extreme VFI network, called XVFI-Net, that first handles the VFI for 4K videos with large motion. The XVFI-Net is based on a recursive multi-scale shared structure that consists of two cascaded modules for bidirectional optical flow learning between two input frames (BiOF-I) and for bidirectional optical flow learning from target to input frames (BiOF-T). The optical flows are stably approximated by a complementary flow reversal (CFR) proposed in BiOF-T module. During inference, the BiOF-I module can start at any scale of input while the BiOF-T module only operates at the original input scale so that the inference can be accelerated while maintaining highly accurate VFI performance. Extensive experimental results show that our XVFI-Net can successfully capture the essential information of objects with extremely large motions and complex textures while the state-of-the-art methods exhibit poor performance. Furthermore, our XVFI-Net framework also performs comparably on the previous lower resolution benchmark dataset, which shows a robustness of our algorithm as well. All source codes, pre-trained models, and proposed X4K1000FPS datasets are publicly available at https://github.com/JihyongOh/XVFI.

Properties of the video frame interpolation models.

Video Interpolation Models. Our experiments use seven advanced deep learning-based VIF methods with various model sizes and properties, including RRIN {{cite:79456ed}}, FLAVR {{cite:bfe453d}}, QVI {{cite:26929b6}}, CAIN {{cite:d6697f7}}, CDFI {{cite:68092ce}}, AdaCoF {{cite:b9ed376}} and XVFI {{cite:26929b6}}. QVI is a flow-based model, while FLAVR, CAIN, CDFI and AdaCoF are kernel-based. And XVFI is the first method proposed for 4K videos with large motion. In the experiments, we employ the pre-trained models provided by original authors. Table {{table:9fd1a9d9-451a-4345-b050-f8844a8970bf}}  shows their characteristics in terms of the number of model parameters, the number of convolutional layers, and class according to {{cite:8727545}}.


Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Abstract: State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available

R-FCN: Object Detection via Region-based Fully Convolutional Networks

Abstract: We present region-based, fully convolutional networks for accurate and efficient object detection. In contrast to previous region-based detectors such as Fast/Faster R-CNN that apply a costly per-region subnetwork hundreds of times, our region-based detector is fully convolutional with almost all computation shared on the entire image. To achieve this goal, we propose position-sensitive score maps to address a dilemma between translation-invariance in image classification and translation-variance in object detection. Our method can thus naturally adopt fully convolutional image classifier backbones, such as the latest Residual Networks (ResNets), for object detection. We show competitive results on the PASCAL VOC datasets (e.g., 83.6% mAP on the 2007 set) with the 101-layer ResNet. Meanwhile, our result is achieved at a test-time speed of 170ms per image, 2.5-20 times faster than the Faster R-CNN counterpart. Code is made publicly available at: https://github.com/daijifeng001/r-fcn.

Searching for MobileNetV3

Abstract: We present the next generation of MobileNets based on a combination of complementary search techniques as well as a novel architecture design. MobileNetV3 is tuned to mobile phone CPUs through a combination of hardware-aware network architecture search (NAS) complemented by the NetAdapt algorithm and then subsequently improved through novel architecture advances. This paper starts the exploration of how automated search algorithms and network design can work together to harness complementary approaches improving the overall state of the art. Through this process we create two new MobileNet models for release: MobileNetV3-Large and MobileNetV3-Small which are targeted for high and low resource use cases. These models are then adapted and applied to the tasks of object detection and semantic segmentation. For the task of semantic segmentation (or any dense pixel prediction), we propose a new efficient segmentation decoder Lite Reduced Atrous Spatial Pyramid Pooling (LR-ASPP). We achieve new state of the art results for mobile classification, detection and segmentation. MobileNetV3-Large is 3.2% more accurate on ImageNet classification while reducing latency by 20% compared to MobileNetV2. MobileNetV3-Small is 6.6% more accurate compared to a MobileNetV2 model with comparable latency. MobileNetV3-Large detection is over 25% faster at roughly the same accuracy as MobileNetV2 on COCO detection. MobileNetV3-Large LR-ASPP is 34% faster than MobileNetV2 R-ASPP at similar accuracy for Cityscapes segmentation.

Objects as Points

Abstract: Detection identifies objects as axis-aligned boxes in an image. Most successful object detectors enumerate a nearly exhaustive list of potential object locations and classify each. This is wasteful, inefficient, and requires additional post-processing. In this paper, we take a different approach. We model an object as a single point --- the center point of its bounding box. Our detector uses keypoint estimation to find center points and regresses to all other object properties, such as size, 3D location, orientation, and even pose. Our center point based approach, CenterNet, is end-to-end differentiable, simpler, faster, and more accurate than corresponding bounding box based detectors. CenterNet achieves the best speed-accuracy trade-off on the MS COCO dataset, with 28.1% AP at 142 FPS, 37.4% AP at 52 FPS, and 45.1% AP with multi-scale testing at 1.4 FPS. We use the same approach to estimate 3D bounding box in the KITTI benchmark and human pose on the COCO keypoint dataset. Our method performs competitively with sophisticated multi-stage methods and runs in real-time.

Overview of Pedestrian Detection Models.

Performance characterization was carried out on seven popular pedestrian detection models. All models use publicly available pretrained weights from two popular model zoos {{cite:12f40b4}}{{cite:b200547}} and are trained using the COCO “train 2017” dataset {{cite:144339a}}. An overview of the pedestrian detection models can be seen in Table {{table:666b5400-f9ad-4d07-8f86-11b338213163}} .
The pedestrian detection models chosen for characterization can be divided into 3 categories: Two-Stage Frameworks, One-Stage Frameworks and Keypoint Estimation.
Two-stage frameworks such as FasterRCNN{{cite:958bbc2}}, MaskRCNN{{cite:adbaee4}} and R-FCN{{cite:c73cc81}} apply two separate networks to perform classification. One network is used to propose regions of interest and a dedicated second network performs object detection {{cite:b13a722}}. One-stage frameworks such as RetinaNet{{cite:d37123d}}, SSD{{cite:eeaa4a1}} and SSDLite{{cite:ad51e82}}{{cite:b19d933}} attempt to reduce computation and increase speed by performing object detection using a single feed forward convolutional network that does not interact with a region proposal module. RetinaNet also implements a novel method of “focal loss” which is used to reduce the imbalance between foreground and background classes during training with a view to increasing detection precision.
CenterNet{{cite:56770fc}} takes an alternative approach based on keypoint estimation. Objects are represented as a single point at their bounding box center identified by a heat map generated using a fully convolutional network. Other object features such as object size, orientation and pose are then regressed directly from the image features at the center location. CenterNet has been shown to outperform a number of state of the art one-stage and two-stage algorithms in terms of a speed-accuracy trade off by maintaining an efficient network architecture{{cite:56770fc}}.


Four factors associated with efficiency loss in the fragmented ridesharing services are identified.

Competition among multiple operators would benefit consumers but lead to a significant operational cost loss.

Trip density, sharing thresholds, market share, spatial unevenness

Economies and Diseconomies of Scale in Segmented Mobility Sharing Markets

Abstract: On-demand mobility sharing, provided by one or several transportation network companies (TNCs), is realized by real-time optimization algorithms to connect trips among tens of thousands of drivers and fellow passengers. In a market of mobility sharing comprised of TNCs, there are two competing principles, the economies of network scale and the healthy competition between TNCs, which can lead to"segmentation"of market. To understand the substantiality and relationship of the two competing principles, we need to answer how much efficiency loss is generated due to the segmentation of market, and which factors are related to it. Here we show how four critical factors of market structure and characteristics of mobility sharing services -- density of trips (thickness), maximum detour allowed for sharing (tightness), market shares (unevenness), and spatial segregation of the TNCs (dissolvedness) -- are associated with the efficiency loss, represented as the difference in vehicle miles traveled (VMT) under different market structures. We found that 1) while VMT shows a simple power function with thickness, the corresponding exponent term can be expressed as a non-monotonic function with tightness -- essentially showing how economies and diseconomies of scale in this market arise, and appearing a very similar form to the Lennard--Jones model in inter-molecular potentials; and 2) the efficiency loss is higher when unevenness is closer to 0.5 (50-50 market share) and dissolvedness is larger. Our results give a comprehensive analysis of how the inefficiency of market segmentation is generated, and how potentially it may be avoided through market mechanism design.

The Price of Fragmentation in Mobility-on-Demand Services

Abstract: Mobility-on-Demand platforms are a fast growing component of the urban transit ecosystem. Though a growing literature addresses the question of how to make individual MoD platforms more efficient, much less is known about the cost of market fragmentation, i.e., the impact on welfare due to splitting the demand between multiple independent platforms. Our work aims to quantify how much platform fragmentation degrades the efficiency of the system. In particular, we focus on a setting where demand is exogenously split between multiple platforms, and study the increase in the supply rebalancing cost incurred by each platform to meet this demand, vis-a-vis the cost incurred by a centralized platform serving the aggregate demand. We show under a large-market scaling, this Price-of-Fragmentation undergoes a phase transition, wherein, depending on the nature of the exogenous demand, the additional cost due to fragmentation either vanishes or grows unbounded. We provide conditions that characterize which regime applies to any given system, and discuss implications of this on how such platforms should be regulated.

Summary of previous studies on inefficiency incurred in fragmented markets

Previous works on the inefficiency of fragmented transportation markets are summarized in Table {{table:701066f0-69da-42d9-a0dd-b0d586cabd99}} . Existing methods adopted in these studies can be generally categorized into two types. On the one hand, market equilibrium models are often used to simulate the interaction between supply and demand of ride-hailing services, based on which we may evaluate the inefficiency from the aspects of platform profits, consumer surplus, and social welfare. {{cite:9f0307d}} built a demand model to confirm that demand fragmentation across multiple platforms could indeed result in higher operating costs and VMT. {{cite:8a5d3b1}} developed a dynamic equilibrium model to analyze the matching friction, and found that market fragmentation could lead to a lower market thickness and longer waiting and cruising time, generating lower total consumer surplus and lower driver revenue compared to the monopolistic market. {{cite:ab46a83}}, {{cite:46357ec}} compared the system performances in the fragmented and integrated platforms at the Nash equilibrium and social optimum. It showed that the integrated platform contributes to higher total realized demand and lower trip fares. They further quantified the inefficiency of the fragmented market in terms of the ratio of social welfare at social optimum to that at Nash equilibrium and found that it decreases as the number of platforms increases.


Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering

Abstract: Many vision and language tasks require commonsense reasoning beyond data-driven image and natural language processing. Here we adopt Visual Question Answering (VQA) as an example task, where a system is expected to answer a question in natural language about an image. Current state-of-the-art systems attempted to solve the task using deep neural architectures and achieved promising performance. However, the resulting systems are generally opaque and they struggle in understanding questions for which extra knowledge is required. In this paper, we present an explicit reasoning layer on top of a set of penultimate neural network based systems. The reasoning layer enables reasoning and answering questions where additional knowledge is required, and at the same time provides an interpretable interface to the end users. Specifically, the reasoning layer adopts a Probabilistic Soft Logic (PSL) based engine to reason over a basket of inputs: visual relations, the semantic parse of the question, and background ontological knowledge from word2vec and ConceptNet. Experimental analysis of the answers and the key evidential predicates generated on the VQA dataset validate our approach.

Multimodal Logical Inference System for Visual-Textual Entailment

Abstract: A large amount of research about multimodal inference across text and vision has been recently developed to obtain visually grounded word and sentence representations. In this paper, we use logic-based representations as unified meaning representations for texts and images and present an unsupervised multimodal logical inference system that can effectively prove entailment relations between them. We show that by combining semantic parsing and theorem proving, the system can handle semantically complex sentences for visual-textual inference.

Interpretable Multimodal Misinformation Detection with Logic Reasoning

Abstract: Multimodal misinformation on online social platforms is becoming a critical concern due to increasing credibility and easier dissemination brought by multimedia content, compared to traditional text-only information. While existing multimodal detection approaches have achieved high performance, the lack of interpretability hinders these systems' reliability and practical deployment. Inspired by NeuralSymbolic AI which combines the learning ability of neural networks with the explainability of symbolic learning, we propose a novel logic-based neural model for multimodal misinformation detection which integrates interpretable logic clauses to express the reasoning process of the target task. To make learning effective, we parameterize symbolic logical elements using neural representations, which facilitate the automatic generation and evaluation of meaningful logic clauses. Additionally, to make our framework generalizable across diverse misinformation sources, we introduce five meta-predicates that can be instantiated with different correlations. Results on three public datasets (Twitter, Weibo, and Sarcasm) demonstrate the feasibility and versatility of our model.

Neural Module Networks

Abstract: Visual question answering is fundamentally compositional in nature-a question like where is the dog? shares substructure with questions like what color is the dog? and where is the cat? This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions. We describe a procedure for constructing and learning neural module networks, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and uses these structures to dynamically instantiate modular networks (with reusable components for recognizing dogs, classifying colors, etc.). The resulting compound networks are jointly trained. We evaluate our approach on two challenging datasets for visual question answering, achieving state-of-the-art results on both the VQA natural image dataset and a new dataset of complex questions about abstract shapes.

Learning to Reason: End-to-End Module Networks for Visual Question Answering

Abstract: Natural language questions are inherently compositional, and many are most easily answered by reasoning about their decomposition into modular sub-problems. For example, to answer "is there an equal number of balls and boxes?" we can look for balls, look for boxes, count them, and compare the results. The recently proposed Neural Module Network (NMN) architecture implements this approach to question answering by parsing questions into linguistic substructures and assembling question-specific deep networks from smaller modules that each solve one subtask. However, existing NMN implementations rely on brittle off-the-shelf parsers, and are restricted to the module configurations proposed by these parsers rather than learning them from data. In this paper, we propose End-to-End Module Networks (N2NMNs), which learn to reason by directly predicting instance-specific network layouts without the aid of a parser. Our model learns to generate network structures (by imitating expert demonstrations) while simultaneously learning network parameters (using the downstream task loss). Experimental results on the new CLEVR dataset targeted at compositional question answering show that N2NMNs achieve an error reduction of nearly 50% relative to state-of-the-art attentional approaches, while discovering interpretable network architectures specialized for each question.

Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding

Abstract: We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data- and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

Abstract: Grounding object properties and relations in 3D scenes is a prerequisite for a wide range of artificial intelligence tasks, such as visually grounded dialogues and embodied manipulation. However, the variability of the 3D domain induces two fundamental challenges: 1) the expense of labeling and 2) the complexity of 3D grounded language. Hence, essential desiderata for models are to be data-efficient, generalize to different data distributions and tasks with unseen semantic forms, as well as ground complex language semantics (e.g., view-point anchoring and multi-object reference). To address these challenges, we propose NS3D, a neuro-symbolic framework for 3D grounding. NS3D translates language into programs with hierarchical structures by leveraging large language-to-code models. Different functional modules in the programs are implemented as neural networks. Notably, NS3D extends prior neuro-symbolic visual reasoning methods by introducing functional modules that effectively reason about high-arity relations (i.e., relations among more than two objects), key in disambiguating objects in complex 3D scenes. Modular and compositional architecture enables NS3D to achieve state-of-the-art results on the ReferIt3D view-dependence task, a 3D referring expression comprehension benchmark. Importantly, NS3D shows significantly improved performance on settings of data-efficiency and generalization, and demonstrate zero-shot transfer to an unseen 3D question-answering task.

Meta Module Network for Compositional Visual Reasoning

Abstract: Neural Module Network (NMN) exhibits strong interpretability and compositionality thanks to its handcrafted neural modules with explicit multi-hop reasoning capability. However, most NMNs suffer from two critical drawbacks: 1) scalability: customized module for specific function renders it impractical when scaling up to a larger set of functions in complex tasks; 2) generalizability: rigid pre-defined module inventory makes it difficult to generalize to unseen functions in new tasks/domains. To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically. The instance modules are then woven into an execution graph for complex visual reasoning, inheriting the strong explainability and compositionality of NMN. With such a flexible instantiation mechanism, the parameters of instance modules are inherited from the central meta module, retaining the same model complexity as the function set grows, which promises better scalability. Meanwhile, as functions are encoded into the embedding space, unseen functions can be readily represented based on its structural similarity with previously observed ones, which ensures better generalizability. Experiments on GQA and CLEVR datasets validate the superiority of MMN over state-of-the-art NMN designs. Synthetic experiments on held-out unseen functions from GQA dataset also demonstrate the strong generalizability of MMN. Our code and model are released in Github https://github.com/wenhuchen/Meta-Module-Network.

ProTo: Program-Guided Transformer for Program-Guided Tasks

Abstract: Programs, consisting of semantic and structural information, play an important role in the communication between humans and agents. Towards learning general program executors to unify perception, reasoning, and decision making, we formulate program-guided tasks which require learning to execute a given program on the observed task specification. Furthermore, we propose the Program-guided Transformer (ProTo), which integrates both semantic and structural guidance of a program by leveraging cross-attention and masked self-attention to pass messages between the specification and routines in the program. ProTo executes a program in a learned latent space and enjoys stronger representation ability than previous neural-symbolic approaches. We demonstrate that ProTo significantly outperforms the previous state-of-the-art methods on GQA visual reasoning and 2D Minecraft policy learning datasets. Additionally, ProTo demonstrates better generalization to unseen, complex, and human-written programs.

Visual Programming: Compositional visual reasoning without training

Abstract: We present Visprog, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. Visprog avoids the need for any task-specific training. Instead, it uses the incontext learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing subroutines, or python functions to produce intermediate outputs that may be consumed by subsequent parts of the program. We demonstrate the flexibility of VIsPROG on 4 diverse tasks - compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing. We believe neuro-symbolic approaches like Visprog are an exciting avenue to easily and effectively expand the scope of AI systems to serve the long tail of complex tasks that people may wish to perform.

Categorization of symbol explanation methods.

Methods of symbol explanation conduct symbol deduction to facilitate cross-modal reasoning, whereby the deducing processes themselves serve as symbol explanations.
Therefore, the provided explanations can consistently reflect the cross-modal reasoning processes of models.
We summarize the symbol explanation methods mentioned above in Table {{table:d50064e6-6860-4145-84f5-dbe3484e41f9}} . Additionally, we further classify methods of logical inference based on the types of formal systems {{cite:b103c03}}, and classify methods of program based on the types of the program modules.


UCI - Individual household electric power consumption

Fremont, CA 15min Retail building electricity load

UK-DALE Domestic Appliance-Level Electricity dataset

2x Conv + 2x LSTM (Encoder) + 2x LSTM (Decoder) + 2x FCL

Building energy load forecasting using Deep Neural Networks

Abstract: Ensuring sustainability demands more efficient energy management with minimized energy wastage. Therefore, the power grid of the future should provide an unprecedented level of flexibility in energy management. To that end, intelligent decision making requires accurate predictions of future energy demand/load, both at aggregate and individual site level. Thus, energy load forecasting have received increased attention in the recent past. However, it has proven to be a difficult problem. This paper presents a novel energy load forecasting methodology based on Deep Neural Networks, specifically, Long Short Term Memory (LSTM) algorithms. The presented work investigates two LSTM based architectures: 1) standard LSTM and 2) LSTM-based Sequence to Sequence (S2S) architecture. Both methods were implemented on a benchmark data set of electricity consumption data from one residential customer. Both architectures were trained and tested on one hour and one-minute time-step resolution datasets. Experimental results showed that the standard LSTM failed at one-minute resolution data while performing well in one-hour resolution data. It was shown that S2S architecture performed well on both datasets. Further, it was shown that the presented methods produced comparable results with the other deep learning methods for energy forecasting in literature.

Building Energy Consumption Prediction: An Extreme Deep Learning Approach

Abstract: Building energy consumption prediction plays an important role in improving the energy utilization rate through helping building managers to make better decisions. However, as a result of randomness and noisy disturbance, it is not an easy task to realize accurate prediction of the building energy consumption. In order to obtain better building energy consumption prediction accuracy, an extreme deep learning approach is presented in this paper. The proposed approach combines stacked autoencoders (SAEs) with the extreme learning machine (ELM) to take advantage of their respective characteristics. In this proposed approach, the SAE is used to extract the building energy consumption features, while the ELM is utilized as a predictor to obtain accurate prediction results. To determine the input variables of the extreme deep learning model, the partial autocorrelation analysis method is adopted. Additionally, in order to examine the performances of the proposed approach, it is compared with some popular machine learning methods, such as the backward propagation neural network (BPNN), support vector regression (SVR), the generalized radial basis function neural network (GRBFNN) and multiple linear regression (MLR). Experimental results demonstrate that the proposed method has the best prediction performance in different cases of the building energy consumption.

Multi-Step Short-Term Power Consumption Forecasting with a Hybrid Deep Learning Strategy

Abstract: Electric power consumption short-term forecasting for individual households is an important and challenging topic in the fields of AI-enhanced energy saving, smart grid planning, sustainable energy usage and electricity market bidding system design. Due to the variability of each household’s personalized activity, difficulties exist for traditional methods, such as auto-regressive moving average models, machine learning methods and non-deep neural networks, to provide accurate prediction for single household electric power consumption. Recent works show that the long short term memory (LSTM) neural network outperforms most of those traditional methods for power consumption forecasting problems. Nevertheless, two research gaps remain as unsolved problems in the literature. First, the prediction accuracy is still not reaching the practical level for real-world industrial applications. Second, most existing works only work on the one-step forecasting problem; the forecasting time is too short for practical usage. In this study, a hybrid deep learning neural network framework that combines convolutional neural network (CNN) with LSTM is proposed to further improve the prediction accuracy. The original short-term forecasting strategy is extended to a multi-step forecasting strategy to introduce more response time for electricity market bidding. Five real-world household power consumption datasets are studied, the proposed hybrid deep learning neural network outperforms most of the existing approaches, including auto-regressive integrated moving average (ARIMA) model, persistent model, support vector regression (SVR) and LSTM alone. In addition, we show a k-step power consumption forecasting strategy to promote the proposed framework for real-world application usage.

Electric Energy Consumption Prediction by Deep Learning with State Explainable Autoencoder

Abstract: As energy demand grows globally, the energy management system (EMS) is becoming increasingly important. Energy prediction is an essential component in the first step to create a management plan in EMS. Conventional energy prediction models focus on prediction performance, but in order to build an efficient system, it is necessary to predict energy demand according to various conditions. In this paper, we propose a method to predict energy demand in various situations using a deep learning model based on an autoencoder. This model consists of a projector that defines an appropriate state for a given situation and a predictor that forecasts energy demand from the defined state. The proposed model produces consumption predictions for 15, 30, 45, and 60 minutes with 60-minute demand to date. In the experiments with household electric power consumption data for five years, this model not only has a better performance with a mean squared error of 0.384 than the conventional models, but also improves the capacity to explain the results of prediction by visualizing the state with t-SNE algorithm. Despite unsupervised representation learning, we confirm that the proposed model defines the state well and predicts the energy demand accordingly.

Improving Electric Energy Consumption Prediction Using CNN and Bi-LSTM

Abstract: The electric energy consumption prediction (EECP) is an essential and complex task in intelligent power management system. EECP plays a significant role in drawing up a national energy development policy. Therefore, this study proposes an Electric Energy Consumption Prediction model utilizing the combination of Convolutional Neural Network (CNN) and Bi-directional Long Short-Term Memory (Bi-LSTM) that is named EECP-CBL model to predict electric energy consumption. In this framework, two CNNs in the first module extract the important information from several variables in the individual household electric power consumption (IHEPC) dataset. Then, Bi-LSTM module with two Bi-LSTM layers uses the above information as well as the trends of time series in two directions including the forward and backward states to make predictions. The obtained values in the Bi-LSTM module will be passed to the last module that consists of two fully connected layers for finally predicting the electric energy consumption in the future. The experiments were conducted to compare the prediction performances of the proposed model and the state-of-the-art models for the IHEPC dataset with several variants. The experimental results indicate that EECP-CBL framework outperforms the state-of-the-art approaches in terms of several performance metrics for electric energy consumption prediction on several variations of IHEPC dataset in real-time, short-term, medium-term and long-term timespans.

Towards Efficient Electricity Forecasting in Residential and Commercial Buildings: A Novel Hybrid CNN with a LSTM-AE based Framework

Abstract: Due to industrialization and the rising demand for energy, global energy consumption has been rapidly increasing. Recent studies show that the biggest portion of energy is consumed in residential buildings, i.e., in European Union countries up to 40% of the total energy is consumed by households. Most residential buildings and industrial zones are equipped with smart sensors such as metering electric sensors, that are inadequately utilized for better energy management. In this paper, we develop a hybrid convolutional neural network (CNN) with an long short-term memory autoencoder (LSTM-AE) model for future energy prediction in residential and commercial buildings. The central focus of this research work is to utilize the smart meters’ data for energy forecasting in order to enable appropriate energy management in buildings. We performed extensive research using several deep learning-based forecasting models and proposed an optimal hybrid CNN with the LSTM-AE model. To the best of our knowledge, we are the first to incorporate the aforementioned models under the umbrella of a unified framework with some utility preprocessing. Initially, the CNN model extracts features from the input data, which are then fed to the LSTM-encoder to generate encoded sequences. The encoded sequences are decoded by another following LSTM-decoder to advance it to the final dense layer for energy prediction. The experimental results using different evaluation metrics show that the proposed hybrid model works well. Also, it records the smallest value for mean square error (MSE), mean absolute error (MAE), root mean square error (RMSE) and mean absolute percentage error (MAPE) when compared to other state-of-the-art forecasting methods over the UCI residential building dataset. Furthermore, we conducted experiments on Korean commercial building data and the results indicate that our proposed hybrid model is a worthy contribution to energy forecasting.

Different forecasting methods in Short Term Demand Forecasting.

In this section, we review different DL architectures (Table {{table:3c9f0e75-5a79-428f-8d56-7541d69274c3}} ) taken from STLF literature to further select a suitable base model for our FL model.

This section defines our data, metrics, model parameters, and how our FL model operates. This FL model predicts the next hour consumption [kWh] based on the consumption data of the last 12  h. This model uses a standardized dataset. However, we evaluate selected literature models from Table {{table:3c9f0e75-5a79-428f-8d56-7541d69274c3}}  using the same dataset to select the best performing one based on the devised statistical metrics. The overall objective of the models' evaluation is to explore their performance in a decentralized environment as the authors implement them in a centralized manner.

Evaluating and selection of Centralized Deep Learning Models

The base of any AI model is the underlying learning method. Thus, we evaluate the performance of the formerly reviewed methods (Table {{table:3c9f0e75-5a79-428f-8d56-7541d69274c3}} ) to compare them using the metrics previously stated in subsection REF . The comparison allows for the later selection of the model our scenarios will use. Each evaluation uses the dataset explained in subsection REF . The modelsThe models have been implemented following their correspondent articles or the code authors have provided. were trained over 100 epochs.


Likelihood of the model generating the reference text.

Is calculated as the harmonic mean of precision and recall.

Similar to BLEU with the difference that it gives higher weight to more valuable mutual n consecutive words.

Like BERTScore calculates similarity matrix but considers many-to-one word relationships.

Can Large Language Models Be an Alternative to Human Evaluations?

Abstract: Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans. However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms.Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided.In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation.We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation.We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: open-ended story generation and adversarial attacks.We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer.We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.

Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond

Abstract: This paper presents a comprehensive and practical guide for practitioners and end-users working with Large Language Models (LLMs) in their downstream natural language processing (NLP) tasks. We provide discussions and insights into the usage of LLMs from the perspectives of models, data, and downstream tasks. Firstly, we offer an introduction and brief summary of current language models. Then, we discuss the influence of pre-training data, training data, and test data. Most importantly, we provide a detailed discussion about the use and non-use cases of large language models for various natural language processing tasks, such as knowledge-intensive tasks, traditional natural language understanding tasks, generation tasks, emergent abilities, and considerations for specific tasks. We present various use cases and non-use cases to illustrate the practical applications and limitations of LLMs in real-world scenarios. We also try to understand the importance of data and the specific challenges associated with each NLP task. Furthermore, we explore the impact of spurious biases on LLMs and delve into other essential considerations, such as efficiency, cost, and latency, to ensure a comprehensive understanding of deploying LLMs in practice. This comprehensive guide aims to provide researchers and practitioners with valuable insights and best practices for working with LLMs, thereby enabling the successful implementation of these models in a wide range of NLP tasks. A curated list of practical guide resources of LLMs, regularly updated, can be found at https://github.com/Mooler0410/LLMsPracticalGuide. An LLMs evolutionary tree, editable yet regularly updated, can be found at llmtree.ai.

A global analysis of metrics used for measuring performance in natural language processing

Abstract: Measuring the performance of natural language processing models is challenging. Traditionally used metrics, such as BLEU and ROUGE, originally devised for machine translation and summarization, have been shown to suffer from low correlation with human judgment and a lack of transferability to other tasks and languages. In the past 15 years, a wide range of alternative metrics have been proposed. However, it is unclear to what extent this has had an impact on NLP benchmarking efforts. Here we provide the first large-scale cross-sectional analysis of metrics used for measuring performance in natural language processing. We curated, mapped and systematized more than 3500 machine learning model performance results from the open repository ‘Papers with Code’ to enable a global and comprehensive analysis. Our results suggest that the large majority of natural language processing metrics currently used have properties that may result in an inadequate reflection of a models’ performance. Furthermore, we found that ambiguities and inconsistencies in the reporting of metrics may lead to difficulties in interpreting and comparing model performances, impairing transparency and reproducibility in NLP research.

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

Abstract: A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.

A brief overview of intrinsic metrics for LLMs

The general automatic evaluation metrics constitute a crucial approach to assessing the performance and quality of LLMs from a linguistic standpoint. These metrics provide quantitative measures that systematically gauge different facets of the language models' proficiency. Notably, the evaluation aspects encompass language fluency, linguistic diversity, coherence, relevance, grammatical correctness, and understanding.
Table {{table:782c89a3-2de5-46d6-9a33-2b5a71c2487a}}  outlines a brief overview of the most commonly used general automatic metrics conventionally employed for LLMs evaluation in the literature.

The purpose of dialogue-based metrics is to provide quantitative assessments of the models' performance capturing various aspects of dialogue quality, including naturalness, informativeness, relevance, and coherence. Primary dialogue-based metrics are Average Request Turn, Match-rate/Success-rate, and Dialogue/Diagnosis Accuracy for healthcare chatbots {{cite:76dabb6}}, {{cite:eb245ac}}, {{cite:3a79e78}}, {{cite:150aba3}}, {{cite:9ca37b7}}. Table {{table:782c89a3-2de5-46d6-9a33-2b5a71c2487a}}  shows a summary of what each dialogue-based metric is measuring.

Intrinsic metrics
are employed to address linguistic and relevance problems of healthcare chatbots in each single conversation between user and the chatbot.
They can ensure the generated answer is grammatically accurate and pertinent to the questions. Table {{table:782c89a3-2de5-46d6-9a33-2b5a71c2487a}}  summarizes the intrinsic metrics used to evaluate LLMs.