How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language

Abstract: One of the factors that have hindered progress in the areas of sign language recognition, translation, and production is the absence of large annotated datasets. Towards this end, we introduce How2Sign, a multimodal and multiview continuous American Sign Language (ASL) dataset, consisting of a parallel corpus of more than 80 hours of sign language videos and a set of corresponding modalities including speech, English transcripts, and depth. A three-hour subset was further recorded in the Panoptic studio enabling detailed 3D pose estimation. To evaluate the potential of How2Sign for real-world impact, we conduct a study with ASL signers and show that synthesized videos using our dataset can indeed be understood. The study further gives insights on challenges that computer vision should address in order to make progress in this field.

MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language

Abstract:
Sign language recognition is a challenging and often underestimated problem comprising multi-modal articulators (handshape, orientation, movement, upper body and face) that integrate asynchronously on multiple streams. Learning powerful statistical models in such a scenario requires much data, particularly to apply recent advances of the field. However, labeled data is a scarce resource for sign language due to the enormous cost of transcribing these unwritten languages.
We propose the first real-life large-scale sign language data set comprising over 25,000 annotated videos, which we thoroughly evaluate with state-of-the-art methods from sign and related action recognition. Unlike the current state-of-the-art, the data set allows to investigate the generalization to unseen individuals (signer-independent test) in a realistic setting with over 200 signers. Previous work mostly deals with limited vocabulary tasks, while here, we cover a large class count of 1000 signs in challenging and unconstrained real-life recording conditions. We further propose I3D, known from video classifications, as a powerful and suitable architecture for sign language recognition, outperforming the current state-of-the-art by a large margin. The data set is publicly available to the community.

Fingerspelling Recognition in the Wild With Iterative Visual Attention

Abstract: Sign language recognition is a challenging gesture sequence recognition problem, characterized by quick and highly coarticulated motion. In this paper we focus on recognition of fingerspelling sequences in American Sign Language (ASL) videos collected in the wild, mainly from YouTube and Deaf social media. Most previous work on sign language recognition has focused on controlled settings where the data is recorded in a studio environment and the number of signers is limited. Our work aims to address the challenges of real-life data, reducing the need for detection or segmentation modules commonly used in this domain. We propose an end-to-end model based on an iterative attention mechanism, without explicit hand detection or segmentation. Our approach dynamically focuses on increasingly high-resolution regions of interest. It out-performs prior work by a large margin. We also introduce a newly collected data set of crowdsourced annotations of fingerspelling in the wild, and show that performance can be further improved with this additional data set.

American Sign Language Fingerspelling Recognition in the Wild

Abstract: We address the problem of American Sign Language fingerspelling recognition “in the wild”, using videos collected from websites. We introduce the largest data set available so far for the problem of fingerspelling recognition, and the first using naturally occurring video data. Using this data set, we present the first attempt to recognize fingerspelling sequences in this challenging setting. Unlike prior work, our video data is extremely challenging due to low frame rates and visual variability. To tackle the visual challenges, we train a special-purpose signing hand detector using a small subset of our data. Given the hand detector output, a sequence model decodes the hypothesized fingerspelled letter sequence. For the sequence model, we explore attention-based recurrent encoder-decoders and CTC-based approaches. As the first attempt at fingerspelling recognition in the wild, this work is intended to serve as a baseline for future work on sign language recognition in realistic conditions. We find that, as expected, letter error rates are much higher than in previous work on more controlled data, and we analyze the sources of error and effects of model variants.

ASL-Homework-RGBD Dataset: An annotated dataset of 45 fluent and non-fluent signers performing American Sign Language homeworks

Abstract: We are releasing a dataset containing videos of both fluent and non-fluent signers using American Sign Language (ASL), which were collected using a Kinect v2 sensor. This dataset was collected as a part of a project to develop and evaluate computer vision algorithms to support new technologies for automatic detection of ASL fluency attributes. A total of 45 fluent and non-fluent participants were asked to perform signing homework assignments that are similar to the assignments used in introductory or intermediate level ASL courses. The data is annotated to identify several aspects of signing including grammatical features and non-manual markers. Sign language recognition is currently very data-driven and this dataset can support the design of recognition technologies, especially technologies that can benefit ASL learners. This dataset might also be interesting to ASL education researchers who want to contrast fluent and non-fluent signing.

Examples of published ASL datasets, with the year of release and the type of signing it contains (Isolated, Continuous, or Fingerspelling). The table indicates whether multiple camera views (e.g., front and side) were included (Multi), whether 3D human skeleton information is included (Pose), whether RGBD depth information is included (Depth), the number of videos (Samples), the number of people in the dataset (Signers), whether the signers were fluent, non-fluent, or both (F/nF), and a citation (Publication). The last row describes the ASL-Homework-RGBD dataset shared with this paper. The non-fluent (“nF") participants in the COPYCAT dataset included Deaf children with developing ASL skills.

Existing datasets of ASL usually consist of videos of people performing continuous signs {{cite:d8e4138}}, {{cite:0d79e85}}, e.g. How2Sign {{cite:25dfd48}}, NCSGLR {{cite:f49466d}}, ASLG-PC{{formula:f290ec31-e943-46a1-b87d-cd91c2ed22ea}}  {{cite:5cf1088}}, CopyCat {{cite:fae9c78}}, RWTH-BOSTON-400 and RWTH-BOSTON-104 {{cite:5f60c49}}, {{cite:6c233c3}}. There are some datasets of isolated signs, e.g. ASL-LEX-2.0 {{cite:0560d4e}}, WLASL {{cite:073d228}}, ASL-100-RGBD {{cite:2205142}}, MSASL {{cite:c012339}}, ASL-LEX {{cite:d0264ab}}, ASLLVD {{cite:8e33af0}}, Purdue RVL-SLL {{cite:73cb281}}, etc., and fingerspelling as well, e.g. ChicagoFSWild+ {{cite:a579067}} and ChicagoFSWild {{cite:fade40e}} . Table {{table:32cb89f1-a7dd-471a-8e4d-1ee67467db56}}
describes some of these datasets in greater detail Table 2 in {{cite:0d79e85}} enlists summary statistics of other sign language datasets..


Abstract: Automatically describing an image with a sentence is a long-standing challenge in computer vision and natural language processing. Due to recent progress in object detection, attribute classification, action recognition, etc., there is renewed interest in this area. However, evaluating the quality of descriptions has proven to be challenging. We propose a novel paradigm for evaluating image descriptions that uses human consensus. This paradigm consists of three main parts: a new triplet-based method of collecting human annotations to measure consensus, a new automated metric that captures consensus, and two new datasets: PASCAL-50S and ABSTRACT-50S that contain 50 sentences describing each image. Our simple metric captures human judgment of consensus better than existing metrics across sentences generated by various sources. We also evaluate five state-of-the-art image description approaches using this new protocol and provide a benchmark for future comparisons. A version of CIDEr named CIDEr-D is available as a part of MS COCO evaluation server to enable systematic evaluation and benchmarking.

Answers Unite! Unsupervised Metrics for Reinforced Summarization Models

Abstract: Abstractive summarization approaches based on Reinforcement Learning (RL) have recently been proposed to overcome classical likelihood maximization. RL enables to consider complex, possibly non differentiable, metrics that globally assess the quality and relevance of the generated outputs. ROUGE, the most used summarization metric, is known to suffer from bias towards lexical similarity as well as from sub-optimal accounting for fluency and readability of the generated abstracts. We thus explore and propose alternative evaluation measures: the reported human-evaluation analysis shows that the proposed metrics, based on Question Answering, favorably compare to ROUGE – with the additional property of not requiring reference summaries. Training a RL-based model on these metrics leads to improvements (both in terms of human or automated metrics) over current approaches that use ROUGE as reward.

BLEURT: Learning Robust Metrics for Text Generation

Abstract: Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgment. We propose BLEURT, a learned evaluation metric for English based on BERT. BLEURT can model human judgment with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG data set. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.

COMET: A Neural Framework for MT Evaluation

Abstract: We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human-mediated Translation Edit Rate and Multidimensional Quality Metrics. Our models achieve new state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems.

Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing

Abstract: We propose the use of a sequence-to-sequence paraphraser for automatic machine translation evaluation. The paraphraser takes a human reference as input and then force-decodes and scores an MT system output. We propose training the aforementioned paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot "language pair" (e.g., Russian to Russian). We denote our paraphraser "unbiased" because the mode of our model's output probability is centered around a copy of the input sequence, which in our case represent the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT19 segment-level shared metrics task in all languages, excluding Gujarati where the model had no training data. We also explore using our model conditioned on the source instead of the reference, and find that it outperforms every quality estimation as a metric system from the WMT19 shared task on quality estimation by a statistically significant margin in every language pair.

CLIPScore: A Reference-free Evaluation Metric for Image Captioning

Abstract: Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality. In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE. Information gain experiments demonstrate that CLIPScore, with its tight focus on image-text compatibility, is complementary to existing reference-based metrics that emphasize text-text similarities. Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation. Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based metrics, e.g., news captions that require richer contextual knowledge.

Transparent Human Evaluation for Image Captioning

Abstract: We establish THumB, a rubric-based human evaluation protocol for image captioning models. Our scoring rubrics and their definitions are carefully developed based on machine- and human-generated captions on the MSCOCO dataset. Each caption is evaluated along two main dimensions in a tradeoff (precision and recall) as well as other aspects that measure the text quality (fluency, conciseness, and inclusive language). Our evaluations demonstrate several critical problems of the current evaluation practice. Human-generated captions show substantially higher quality than machine-generated ones, especially in coverage of salient information (i.e., recall), while most automatic metrics say the opposite. Our rubric-based results reveal that CLIPScore, a recent metric that uses image features, better correlates with human judgments than conventional text-only metrics because it is more sensitive to recall. We hope that this work will promote a more transparent evaluation protocol for image captioning and its automatic metrics.

We thank Kyunghyun Cho, Elizabeth Clark, Jesse Dodge, Saadia Gabriel, Michal Guerquin, Daniel Khashabi, Jamie Morgenstern, Phoebe Mulcaire, Hao Peng, Sofia Serrano, the Ark group at UW, and the Mosaic team at Allen Institute for AI for their helpful feedback on this work.
We also thank Xinyan Yu for WMT Chinese-English meta-evaluations and Muyun Yang and Shujian Huang for sharing with us the CWMT training data.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.
SPICE: semantic
propositional image caption evaluation.
In Proc. of ECCV.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen
Gould, and Lei Zhang. 2018.
Bottom-up and top-down
attention for image captioning and visual question answering.
In Proc. of CVPR.
Satanjeev Banerjee and Alon Lavie. 2005.
METEOR: An automatic
metric for MT evaluation with improved correlation with human judgments.
In Proc. of the ACL Workshop on Intrinsic and Extrinsic
Evaluation Measures for Machine Translation and/or Summarization.
Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R.
Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz,
Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu
Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki
Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020.
Findings of the 2020
conference on machine translation (WMT20).
In Proc. of WMT.
Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015.
Fitting linear mixed-effects models using lme4.
Journal of Statistical Software.
Rachel Bawden, Giorgio Maria Di Nunzio, Cristian Grozea, Inigo Jauregi Unanue,
Antonio Jimeno Yepes, Nancy Mah, David Martinez, Aurélie Névéol,
Mariana Neves, Maite Oronoz, Olatz Perez-de Viñaspre, Massimo Piccardi,
Roland Roller, Amy Siu, Philippe Thomas, Federica Vezzani, Maika
Vicente Navarro, Dina Wiemann, and Lana Yeganova. 2020.
Findings of the WMT
2020 biomedical translation shared task: Basque, Italian and Russian as
new additional languages.
In Proc. of WMT.
Steven Bird, Evan Klein, and Edward Loper. 2009.
Natural
Language Processing with Python.
Cambridge University Press.
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and
Iryna Gurevych. 2019.
Better rewards yield better
summaries: Learning to summarise without references.
In Proc. of EMNLP.
Léo Bouscarrat, Antoine Bonnefoy, Thomas Peel, and Cécile Pereira.
2019.
STRASS: A light and
effective method for extractive summarization based on sentence embeddings.
In Proc. of ACL.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon
Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris
Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever,
and Dario Amodei. 2020.
Language models are few-shot learners.
In Proc. of NeurIPS.
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh
Schroeder. 2007.
(meta-) evaluation of
machine translation.
In Proc. of WMT.
Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh
Schroeder. 2008.
Further
meta-evaluation of machine translation.
In Proc. of WMT.
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006.
Re-evaluating the
role of Bleu in machine translation research.
In Proc. of EACL.
Tanfang Chen, Weiwei Wang, Wenyang Wei, Xing Shi, Xiangang Li, Jieping Ye, and
Kevin Knight. 2020.
DiDi's machine
translation system for WMT2020.
In Proc. of WMT.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta,
Piotr Dollár, and C. Lawrence Zitnick. 2015.
Microsoft COCO captions:
Data collection and evaluation server.
Yen-Chun Chen and Mohit Bansal. 2018.
Fast abstractive
summarization with reinforce-selected sentence rewriting.
In Proc. of ACL.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan,
and Noah A. Smith. 2021.
All that's `human' is
not gold: Evaluating human evaluation of generated text.
In Proc. of ACL.
Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. 2019.
Sentence mover's
similarity: Automatic evaluation for multi-sentence texts.
In Proc. of ACL.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume
Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and
Veselin Stoyanov. 2020.
Unsupervised cross-lingual
representation learning at scale.
In Proc. of ACL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.
BERT: Pre-training of deep
bidirectional transformers for language understanding.
In Proc. of NAACL.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao,
Ming Zhou, and Hsiao-Wuen Hon. 2019.
Unified language model
pre-training for natural language understanding and generation.
In Proc. of NeurIPS.
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit
Cheung. 2018.
BanditSum: Extractive
summarization as a contextual bandit.
In Proc. of EMNLP.
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato, and Michael Auli. 2020.
On the evaluation of
machine translation systems trained with back-translation.
In Proc. of ACL.
Kawin Ethayarajh and Dan Jurafsky. 2020.
Utility is in the eye of
the user: A critique of NLP leaderboards.
In Proc. of EMNLP.
Matan Eyal, Tal Baumel, and Michael Elhadad. 2019.
Question answering as an
automatic evaluation metric for news article summarization.
In Proc. of NAACL.
Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong,
Richard Socher, and Dragomir Radev. 2021.
SummEval: Re-evaluating
summarization evaluation.
TACL.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and
Wolfgang Macherey. 2021.
Experts, errors, and
context: A large-scale study of human evaluation for machine translation.
Markus Freitag, David Grangier, and Isaac Caswell. 2020.
BLEU might be guilty but
references are not innocent.
In Proc. of EMNLP.
Sebastian Gehrmann, Tosin P. Adewumi, Karmanya Aggarwal, Pawan Sasanka
Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu,
Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh D. Dhole, Wanyu Du, Esin
Durmus, Ondrej Dusek, Chris Emezue, Varun Gangal, Cristina Garbacea,
Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji,
Shailza Jolly, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela,
Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique
Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin
Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey
Osei, Ankur P. Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas
Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault
Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla
Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila
Yerukola, and Jiawei Zhou. 2021.
The GEM benchmark:
Natural language generation, its evaluation and metrics.
In Proc. of GEM.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018.
Bottom-up abstractive
summarization.
In Proc. of EMNLP.
Ulrich Germann. 2020.
The University of
Edinburgh's submission to the German-to-English and
English-to-German tracks in the WMT 2020 news translation and zero-shot
translation robustness tasks.
In Proc. of WMT.
Dan Gillick and Yang Liu. 2010.
Non-expert evaluation of
summarization systems is risky.
In Proc. of the NAACL HLT 2010 Workshop on Creating Speech
and Language Data with Amazon's Mechanical Turk.
Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington,
Kris M. Kitani, and Jeffrey P. Bigham. 2020.
Twitter
a11y: A browser extension to make twitter images accessible.
In Proc. of CHI.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018.
Soft layer-specific
multi-task summarization with entailment and question generation.
In Proc. of ACL.
Jeremy Gwinnup and Tim Anderson. 2020.
The AFRL WMT20
news translation systems.
In Proc. of WMT.
Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark,
Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis,
Mengnan Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin,
Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong
Zhang, Zhirui Zhang, and Ming Zhou. 2018.
Achieving human parity on
automatic Chinese to English news translation.
Kenneth Heafield, Hiroaki Hayashi, Yusuke Oda, Ioannis Konstas, Andrew Finch,
Graham Neubig, Xian Li, and Alexandra Birch. 2020.
Findings of the fourth
workshop on neural generation and translation.
In Proc. of WNGT.
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse
Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.
Teaching machines to read
and comprehend.
In Proc. of NeurIPS.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021.
CLIPScore: A
reference-free evaluation metric for image captioning.
In Proc. of EMNLP.
Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun.
2018.
A unified model for
extractive and abstractive summarization using inconsistency loss.
In Proc. of ACL.
Yichen Jiang and Mohit Bansal. 2018.
Closed-book training to
improve summarization encoder memory.
In Proc. of EMNLP.
Andrej Karpathy and Li Fei-Fei. 2015.
Deep visual-semantic
alignments for generating image descriptions.
In Proc. of CVPR.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A. Smith.
2021a.
Deep encoder, shallow
decoder: Reevaluating non-autoregressive machine translation.
In Proc. of ICLR.
Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos
Pappas, Yi Mao, Weizhu Chen, and Noah A. Smith. 2021b.
Finetuning pretrained
transformers into RNNs.
In Proc. of EMNLP.
Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras,
Yejin Choi, and Noah A. Smith. 2021c.
Transparent human
evaluation for image captioning.
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo
Kasai, Yejin Choi, Noah A. Smith, and Daniel S. Weld. 2021.
GENIE: A leaderboard
for human-in-the-loop evaluation of text generation.
Shun Kiyono, Takumi Ito, Ryuto Konno, Makoto Morishita, and Jun Suzuki. 2020.
Tohoku-AIP-NTT at
WMT 2020 news translation task.
In Proc. of WMT.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello
Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard
Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst.
2007.
Moses: Open source
toolkit for statistical machine translation.
In Proc. of ACL Demo and Poster Sessions.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and
Richard Socher. 2019.
Neural text summarization:
A critical evaluation.
In Proc. of EMNLP.
Wojciech Kryściński, Romain Paulus, Caiming Xiong, and Richard Socher.
2018.
Improving abstraction in
text summarization.
In Proc. of EMNLP.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.
BART: Denoising
sequence-to-sequence pre-training for natural language generation,
translation, and comprehension.
In Proc. of ACL.
Zuchao Li, Hai Zhao, Rui Wang, Kehai Chen, Masao Utiyama, and Eiichiro Sumita.
2020.
SJTU-NICT's
supervised and unsupervised neural machine translation systems for the
WMT20 news translation task.
In Proc. of WMT.
Chin-Yew Lin. 2004.
ROUGE: A
package for automatic evaluation of summaries.
In Proc. of Text Summarization Branches Out.
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B.
Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and
C. Lawrence Zitnick. 2014.
Microsoft COCO: common
objects in context.
In Proc. of ECCV.
Feifan Liu and Yang Liu. 2008.
Correlation between
ROUGE and human evaluation of extractive meeting summaries.
In Proc. of ACL.
Yang Liu and Mirella Lapata. 2019.
Text summarization with
pretrained encoders.
In Proc. of EMNLP.
Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019.
Results of the
WMT19 metrics shared task: Segment-level and strong MT systems pose big
challenges.
In Proc. of WMT.
Matouš Macháček and Ondřej Bojar. 2014.
Results of the WMT14
metrics shared task.
In Proc. of WMT.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J.
Bethard, and David McClosky. 2014.
The
Stanford CoreNLP natural language processing toolkit.
In Proc. of ACL System Demonstrations.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a.
Tangled up in BLEU:
Reevaluating the evaluation of automatic machine translation evaluation
metrics.
In Proc. of ACL.
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar.
2020b.
Results of the
WMT20 metrics shared task.
In Proc. of WMT.
Fandong Meng, Jianhao Yan, Yijin Liu, Yuan Gao, Xianfeng Zeng, Qinsong Zeng,
Peng Li, Ming Chen, Jie Zhou, Sifan Liu, and Hao Zhou. 2020.
WeChat neural machine
translation systems for WMT20.
In Proc. of WMT.
Alexander Molchanov. 2020.
PROMT systems for
WMT 2020 shared news translation task.
In Proc. of WMT.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar
Gülçehre, and Bing Xiang. 2016.
Abstractive text
summarization using sequence-to-sequence RNNs and beyond.
In Proc. of CoNLL.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.
Ranking sentences for
extractive summarization with reinforcement learning.
In Proc. of NAACL.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov.
2019.
Facebook FAIR's
WMT19 news translation task submission.
In Proc. of WMT.
Csaba Oravecz, Katina Bontcheva, László Tihanyi, David Kolovratnik,
Bhavani Bhaskar, Adrien Lardilleux, Szymon Klocek, and Andreas Eisele. 2020.
eTranslation's
submissions to the WMT 2020 news translation task.
In Proc. of WMT.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng,
David Grangier, and Michael Auli. 2019.
fairseq: A fast, extensible
toolkit for sequence modeling.
In NAACL Demonstrations.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.
BLEU: a method
for automatic evaluation of machine translation.
In Proc. of ACL.
Ramakanth Pasunuru and Mohit Bansal. 2018.
Multi-reward reinforced
summarization with saliency and entailment.
In Proc. of NAACL.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and
Lingpeng Kong. 2021.
Random feature attention.
In Proc. of ICLR.
Maja Popović. 2015.
chrF: character n-gram
F-score for automatic MT evaluation.
In Proc. of WMT.
Maja Popović. 2017.
chrF++: words helping
character n-grams.
In Proc. of WMT.
Matt Post. 2018.
A call for clarity
in reporting BLEU scores.
In Proc. of WMT.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.
Exploring the limits
of transfer learning with a unified text-to-text transformer.
JLMR, 21.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.
SQuAD: 100,000+
questions for machine comprehension of text.
In Proc. of EMNLP.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.
COMET: A neural framework
for MT evaluation.
In Proc. of EMNLP.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein,
Alexander C. Berg, and Li Fei-Fei. 2015.
ImageNet large scale
visual recognition challenge.
IJCV.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019.
Green AI.
CACM.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano.
2019.
Answers unite! unsupervised
metrics for reinforced summarization models.
In Proc. of EMNLP.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.
Get to the point:
Summarization with pointer-generator networks.
In Proc. of ACL.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020.
BLEURT: Learning robust
metrics for text generation.
In Proc. of ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a.
Improving neural machine
translation models with monolingual data.
In Proc. of ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b.
Neural machine
translation of rare words with subword units.
In Proc. of ACL.
Eva Sharma, Luyang Huang, Zhe Hu, and Lu Wang. 2019.
An entity-driven framework
for abstractive summarization.
In Proc. of EMNLP.
Tingxun Shi, Shiyu Zhao, Xiaopu Li, Xiaoxue Wang, Qian Zhang, Di Ai, Dawei
Dang, Xue Zhengshan, and Jie Hao. 2020.
OPPO's machine
translation systems for WMT20.
In Proc. of WMT.
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul.
2006.
A study of
translation edit rate with targeted human annotation.
In AMTA.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019.
Evaluating gender bias in
machine translation.
In Proc. of ACL.
Brian Thompson and Matt Post. 2020.
Automatic machine
translation evaluation in many languages via zero-shot paraphrasing.
In Proc. of EMNLP.
Robert Tibshirani. 1994.
Regression shrinkage
and selection via the Lasso.
Journal of the Royal Statistical Society, Series B.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018.
Attaining the unattainable?
reassessing claims of human parity in neural machine translation.
In Proc. of WMT.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.
Attention is all you
need.
In Proc. of NeurIPS.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.
CIDEr: Consensus-based
image description evaluation.
In Proc. of CVPR.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019.
When a good translation is
wrong in context: Context-aware machine translation improves on deixis,
ellipsis, and lexical cohesion.
In Proc. of ACL.
Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016.
CharacTer: Translation
edit rate on character level.
In Proc. of WMT.
Daimeng Wei, Hengchao Shang, Zhanglin Wu, Zhengzhe Yu, Liangyou Li, Jiaxin Guo,
Minghan Wang, Hao Yang, Lizhi Lei, Ying Qin, and Shiliang Sun. 2020.
HW-TSC's
participation in the WMT 2020 news translation shared task.
In Proc. of WMT.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe
Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien
Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
and Alexander Rush. 2020.
HuggingFace's
transformers: State-of-the-art natural language processing.
In Proc. of EMNLP: System Demonstrations.
Liwei Wu, Xiao Pan, Zehui Lin, Yaoming Zhu, Mingxuan Wang, and Lei Li.
2020a.
The Volctrans machine
translation system for WMT20.
In Proc. of WMT.
Shuangzhi Wu, Xing Wang, Longyue Wang, Fangxu Liu, Jun Xie, Zhaopeng Tu,
Shuming Shi, and Mu Li. 2020b.
Tencent neural
machine translation systems for the WMT20 news translation task.
In Proc. of WMT.
Yuxiang Wu and Baotian Hu. 2018.
Learning to extract
coherent summary via deep reinforcement learning.
In Proc. of AAAI.
Jiacheng Xu and Greg Durrett. 2019.
Neural extractive text
summarization with syntactic compression.
In Proc. of EMNLP.
Lei Yu, Laurent Sartran, Po-Sen Huang, Wojciech Stokowiec, Domenic Donato,
Srivatsan Srinivasan, Alek Andreev, Wang Ling, Sona Mokra, Agustin Dal Lago,
Yotam Doron, Susannah Young, Phil Blunsom, and Chris Dyer. 2020.
The DeepMind
Chinese–English document translation system at WMT2020.
In Proc. of WMT.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a.
PEGASUS: Pre-training
with extracted gap-sentences for abstractive summarization.
In Proc. of ICML.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang,
Yejin Choi, and Jianfeng Gao. 2021.
VinVL: Making visual
representations matter in vision-language models.
In CVPR 2021.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi.
2020b.
BERTScore: Evaluating
text generation with BERT.
In Proc. of ICLR.
Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018.
Neural latent extractive
document summarization.
In Proc. of EMNLP.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng
Gao. 2020.
Unified vision-language
pre-training for image captioning and VQA.
In Proc. of AAAI.
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao.
2018.
Neural document
summarization by jointly learning to score and select sentences.
In Proc. of ACL.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford,
Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019.
Fine-tuning language models
from human preferences.

Case Studies of Evaluation Practice
Fig. {{figure:c41807e7-c5d3-475a-bbc1-2cb00d5de292}}  depicts breakdowns of evaluation metrics used in the papers on machine translation and summarization from NAACL and ACL 2021. We examined all papers whose title contains “machine translation” and “summarization.”
We see the clear gap between generation modeling and evaluation research; most researchers do not take advantage of recent metrics that correlate better with human judgments.
Breakdowns of evaluation metrics used in the papers on machine translation and summarization from NAACL and ACL 2021. We examined all papers whose title contains “machine translation” and “summarization” and disregarded papers primarily on evaluation metrics. “QA” metrics use a QA system to evaluate summaries (e.g., {{cite:00c92e9}}). “Specialized” indicates specialized evaluation in a particular dimension, rather than the overall generation quality, such as document-level evaluations on contrastive sets {{cite:25a5803}}.
{{figure:a129e16e-1fd5-42f5-98e9-ac90f924f558}}{{figure:e0f5c027-0276-4f52-82ea-f779d7f3a3c0}}{{figure:c41807e7-c5d3-475a-bbc1-2cb00d5de292}}
Participating Generators
Here we list the generators submitted in the initial Billboards.
WMT20 ZH-EN
Transformer-base fairseq hyperparameters and setting.
[.1em]
Hyperparameter
Value
[.1em]
label smoothing
0.1
# max tokens
1024
dropout rate
0.1
encoder embedding dim
512
encoder ffn dim
2048
# encoder attn heads
8
decoder embedding dim
512
decoder ffn dim
2048
# decoder attn heads
8
max source positions
1024
max target positions
1024
Adam lrate
{{formula:e950ebf9-f2f5-45f6-91b7-b509a5f41300}}
Adam {{formula:3fc63b09-11a9-4366-b750-726a9df8564b}}
0.9
Adam {{formula:fa8507ca-1b37-4f0e-b4f0-c0b3b8691ae2}}
0.98
lr-scheduler
inverse square
warm-up lr
{{formula:db841c45-a51f-46ff-8ec6-797f6f6d8eef}}
# warmup updates
4000
# max updates
600K
# GPUs
8
length penalty
0.6
[.1em]
{{table:9c9925af-1a38-48e1-abc6-7f290673da9f}}{{table:695a8848-1993-4fbf-9211-2e04b47435a2}}Transformer-large and transformer-large-ensemble fairseq hyperparameters and setting. Transformer-large-ensemble ensembles four transformer-large models with different random initializations.
[.1em]
Hyperparameter
Value
[.1em]
label smoothing
0.1
# max tokens
4096
dropout rate
0.1
encoder embedding dim
1024
encoder ffn dim
4096
# encoder attn heads
16
decoder embedding dim
1024
decoder ffn dim
4096
# decoder attn heads
16
max source positions
1024
max target positions
1024
Adam lrate
{{formula:5d9e96ba-58d8-4148-9f33-d62d6f77a427}}
Adam {{formula:67ed223d-da88-44ef-89e0-15ceeb3f7812}}
0.9
Adam {{formula:190bacff-c9f7-46cd-a4b5-c2a1b9782ffb}}
0.98
lr-scheduler
inverse square
warm-up lr
{{formula:721bb4ba-9bcd-4472-ae2a-4e734399590d}}
# warmup updates
4000
# max updates
600K
# GPUs
8
length penalty
0.6
[.1em]
{{table:f40cde66-f87f-41c4-843b-30ce412aa871}}{{table:336176ab-15ec-46b7-a7f8-747d95a9acf8}}We use all 16 submissions for the WMT20 ZH-EN task {{cite:2ae65e9}}https://www.statmt.org/wmt20/results.html. as well as our own three transformer baselines that were implemented in fairseq {{cite:8d623b0}}.
Our baselines allow researchers to compare their translation models without resource-intensive techniques such as backtranslation {{cite:461913f}}, model ensembling, and deep encoders {{cite:4c52c03}}.
Tables {{table:695a8848-1993-4fbf-9211-2e04b47435a2}}  and {{table:336176ab-15ec-46b7-a7f8-747d95a9acf8}}  list the hyperprameters.
We generally follow the setting from {{cite:2619b1d}}.
We use newstest-2019 as the dev. set and the official training data.http://www.statmt.org/wmt20/translation-task.html.
We apply moses tokenization {{cite:2881e33}} and BPE with 32K operations {{cite:3b69875}} to English text.
We tokenize Chinese text with the Jieba package,https://github.com/fxsjy/jieba. following {{cite:bd25404}}.
Separately from English, BPE with 32K operations are then applied to Chinese.
The decoder input and output embeddings are tied.
Moses detokenization is applied to get the final outputs in the last step.
We make the three models and preprocessed train/dev. data publicly available.https://github.com/jungokasai/billboard/tree/master/baselines.
Table {{table:20369ad3-1d66-4998-9e17-4a690ba8534a}}  lists all generators and their automatic evaluation scores from the top-performing metric (ensemble in this case).
WMT20 ZH-EN generators and reference papers. The score column indicates the score from the metric that currently correlates best with the human judgments (ensemble).
[.1em]
Generator
Description
Score
[.1em]
Huoshan Translate
{{cite:163964e}}
78.85
THUNLP
Not available
78.81
Huawei TSC
{{cite:916b3e3}}
78.79
DeepMind
{{cite:e61bfde}}
78.76
WeChat AI
{{cite:f236a10}}
78.75
Tencent Translation
{{cite:dfdc29d}}
78.74
DiDi NLP
{{cite:efedc79}}
78.66
OPPO
{{cite:19bc377}}
78.59
Online-B
Not available
78.36
SJTU-NICT
{{cite:1bb7b5c}}
78.27
trans-large-ensemble
§ REF
77.35
trans-large
§ REF
76.98
Online-A
Not available
76.86
trans-base
§ REF
76.79
dong-nmt
Not available
76.74
Online-G
Not available
76.44
zlabs-nlp
Not available
75.79
Online-Z
Not available
75.05
WMT Biomed Baseline
{{cite:8f7e444}}
73.89
[.1em]
{{table:1688d88f-9549-4ff9-a6fc-6fd030d110f4}}{{table:20369ad3-1d66-4998-9e17-4a690ba8534a}}
WMT20 EN-DE
Similar to WMT20 ZH-EN, we use all 14 submissions for the WMT20 EN-DE task along with our three transformer baselines.
The same hyperparameters are chosen as in WMT20 ZH-EN (Tables {{table:695a8848-1993-4fbf-9211-2e04b47435a2}}  and {{table:336176ab-15ec-46b7-a7f8-747d95a9acf8}} ).
We preprocess both English and German text by Moses tokenizer and joint BPE with 32K operations. All embeddings are shared.
We apply Moses detokenizer to get the final outputs.
Table {{table:46e1251b-104b-4c13-a872-bbb6872ee38d}}  shows the generators and their automatic evaluation scores from the top-performing metric (ensemble).
WMT20 EN-DE generators and reference papers. The score column indicates the score from the metric that currently correlates best with the human judgments (ensemble).
[.1em]
Generator
Description
Score
[.1em]
Tohoku-AIP-NTT
{{cite:40d4e2d}}
90.50
Tencent Translate
{{cite:dfdc29d}}
90.43
OPPO
{{cite:19bc377}}
90.42
eTranslation
{{cite:d6c31e1}}
90.39
Online-B
Not available
90.38
Huoshan Translate
{{cite:163964e}}
90.32
AFRL             {{cite:a0d350f}}
90.16
Online-A
Not available
90.12
UEDIN
{{cite:8fe1291}}
89.98
PROMT NMT
{{cite:48f2001}}
89.66
trans-large
§ REF
89.60
trans-large-ensemble
§ REF
89.59
trans-base
§ REF
89.35
Online-Z
Not available
89.26
Online-G
Not available
88.98
zlabs-nlp
Not available
88.65
WMT Biomed Baseline
{{cite:8f7e444}}
88.23
[.1em]
{{table:b15c3ed0-5082-4a62-8b7a-1228f239c409}}{{table:46e1251b-104b-4c13-a872-bbb6872ee38d}}
CNNDM Summarization
We submit all 26 models from {{cite:26cdf45}}.https://github.com/jungokasai/THumB/tree/master/cnndm.
Table {{table:46e1251b-104b-4c13-a872-bbb6872ee38d}}  shows all models and their automatic evaluation scores from the top-performing metric (COMET).
CNNDM summarization generators and reference papers.
They are from {{cite:26cdf45}}, but we apply detokenization {{cite:79a950f}} and/or truecasing {{cite:8355ca3}} to standardize the model outputs for better, reproducible evaluations.
The score column indicates the score from the metric that currently correlates best with the human judgments (COMET).
[.1em]
Generator
Description
Score
[.1em]
Lead-3
First 3 sentences
-0.011
T5
{{cite:1a77454}}
-0.030
BART
{{cite:b291e3f}}
-0.032
Pegasus-dynamic-mix
{{cite:894f42f}}
-0.044
RNES
{{cite:a7ce401}}
-0.049
Unified-ext-abs
{{cite:2a2da73}}
-0.056
Pegasus-huge-news
{{cite:894f42f}}
-0.056
REFRESH
{{cite:b072508}}
-0.067
ROUGESal             {{cite:f33693f}}
-0.073
Human-H
Highlights
-0.075
NEUSUM
{{cite:e285fc4}}
-0.083
BanditSum
{{cite:f12e95d}}
-0.083
LATENT
{{cite:dd4abc3}}
-0.099
Closed-book-decoder
{{cite:2c073c3}}
-0.112
Multi-task-Ent-QG
{{cite:ee44886}}
-0.117
Pointer-Generator
{{cite:0a3849e}}
-0.144
UniLM
{{cite:03efa7e}}
-0.151
Bottom-Up
{{cite:1e3ff44}}
-0.160
JEC                         {{cite:2095d57}}
-0.167
Fast-abs-rl
{{cite:6082a13}}
-0.189
NeuralTD
{{cite:9ae6ce8}}
-0.215
Improve-abs
{{cite:74143a2}}
-0.329
BertSum-abs
{{cite:8830be2}}
-0.341
STRASS
{{cite:e60252b}}
-0.405
GPT-2-zero-shot
{{cite:77f01e3}}
-0.441
SENECA
{{cite:2aaa5fc}}
-0.735
[.1em]
{{table:0875311a-f90e-47a1-91d9-ec6e19a3925f}}{{table:3f6b54d3-e74a-4a4f-8ef6-70a7cf8f3d20}}
MSCOCO Image Captioning
We submit the four strong models from the literature {{cite:2fe3647}}.https://github.com/jungokasai/THumB/tree/master/mscoco.
They share similar pipeline structure but vary in model architecture, (pre)training data, model size, and (pre)training objective.
Table {{table:8f609a3d-e094-43b3-95bb-cc5cf04e53fb}}  shows the models with their papers and automatic scores from the top-performing metric (RefCLIP-S).
MSCOCO image captioning generators and reference papers. The score column indicates the score from the metric that currently correlates best with the human judgments (RefCLIP-S).
[.1em]
Generator
Description
Score
[.1em]
VinVL-largeModel with CIDEr optmization, https://github.com/microsoft/Oscar/blob/master/VinVL_MODEL_ZOO.md#Image-Captioning-on-COCO.
{{cite:97c52b5}}
83.78
VinVL-baseModel with CIDEr optmization.
{{cite:97c52b5}}
83.45
Unified-VLP
{{cite:912737d}}
82.59
Up-DownModel with cross-entropy optimization, https://vision-explorer.allenai.org/image_captioning.
{{cite:389c750}}
80.63
[.1em]
{{table:0b206699-c7c5-462c-ab65-dca74e2fefda}}{{table:8f609a3d-e094-43b3-95bb-cc5cf04e53fb}}
Participating Metrics
Table {{table:697d2192-0e94-4774-9862-fe4d51dd126d}}  discusses details and configurations of the automatic metrics that we implement in our initial Billboards.
Automatic metrics and their reference papers. The refs., src., and cont. columns indicate whether they use references, input source features, and pretrained contextual representations (e.g., BERT; {{cite:7fc15ce}}), respectively.
[.1em]
Metric
Description
Refs.
Src.
Cont.
[.1em]
BLEUSacreBLEU implementation of sentence-level BLEU-4; https://github.com/mjpost/sacreBLEU/blob/v1.2.12/sacrebleu.py#L999.
{{cite:75cd27a}}
✓
✗
✗
ROUGE-3HuggingFace implementation {{cite:ffba039}}.
{{cite:eb385d1}}
✓
✗
✗
ROUGE-L
{{cite:eb385d1}}
✓
✗
✗
METEOR
{{cite:aedfd8f}}
✓
✗
✗
TERhttps://github.com/mjpost/sacrebleu.
{{cite:ffcbec8}}
✓
✗
✗
METEORhttps://www.nltk.org/_modules/nltk/translate/meteor_score.html.
{{cite:aedfd8f}}
✓
✗
✗
chrFhttps://github.com/m-popovic/chrF.
{{cite:dd54db3}}
✓
✗
✗
CIDErhttps://github.com/salaniz/pycocoevalcap.
{{cite:59d85b3}}
✓
✗
✗
SPICE
{{cite:2923d19}}
✓
✗
✗
CharacTERhttps://github.com/rwth-i6/CharacTER.
{{cite:394caec}}
✓
✗
✗
chrF++
{{cite:b507e32}}
✓
✗
✗
SummaQAhttps://github.com/ThomasScialom/summa-qa.
{{cite:b5b73e0}}
✗
✓
✓
BERTScore
{{cite:52fb126}}
✓
✗
✓
BLEURThttps://huggingface.co/metrics/bleurt.
{{cite:1aaeb5d}}
✓
✗
✓
COMEThttps://github.com/Unbabel/COMET/.
{{cite:46fc863}}
✓
✓
✓
COMET-QE
{{cite:46fc863}}
✗
✓
✓
Prism-refhttps://github.com/thompsonb/prism.
{{cite:6362b3d}}
✓
✗
✓
Prism-src
{{cite:6362b3d}}
✗
✓
✓
CLIP-Shttps://github.com/salaniz/pycocoevalcap.
{{cite:7d73897}}
✗
✓
✓
RefCLIP-S
{{cite:7d73897}}
✓
✓
✓
RefOnlyC
{{cite:2fe3647}}
✓
✗
✓
[.1em]
{{table:02d3d98e-df46-436c-9e18-2b7b5275261b}}{{table:697d2192-0e94-4774-9862-fe4d51dd126d}}
Additional Ensemble Metric Ablations
Seen in Table {{table:9a20cb4f-0052-42a8-b496-bd7ebb2d2df8}}  are ablation studies for the ensemble metrics where one of the three selected metrics is removed at a time.
Dropping one metric often has no impact on the correlation score, suggesting that these metrics are highly redundant and capture similar aspects of the output quality.
Billboards encourage researchers to explore ways to diversify automatic evaluations by updating the ensemble metric every time a new metric is submitted.
Correlations from ensemble ablation studies. One of the three selected metrics is removed at a time, and a new Lasso regression model is trained on the remaining metrics. The bigger the correlation drop is, the bigger the contribution is from the removed metric. magentaCOMET-QE is a referenceless metric.
2*ZH-EN
–
COMET
magentaCOMET-QE
BLEURT

0.61
0.61
0.57
0.61
2*EN-DE
–
COMET
magentaCOMET-QE
Prism-ref

0.51
0.52
0.52
0.52
2*CNNDM
–
COMET
magentaCOMET-QE
BERTScore

0.29
0.23
0.31
0.31
2*COCO
–
RefCLIP-S
RefOnlyC
CIDEr

0.45
0.44
0.42
0.43
{{table:ee4bbd36-e5cb-481b-84f1-78c28c5c744f}}{{table:9a20cb4f-0052-42a8-b496-bd7ebb2d2df8}}
Additional Mixed-Effects Analysis
Table {{table:d7c43f3d-5ce0-4abd-89f6-f52c9a198671}}  presents fixed effect coefficients that measure how much each automatic metric overrates machines over humans (§REF ).
With some exceptions in CNNDM summarization, almost all automatic metrics underrate human generations (significantly positive coefficients).
Table {{table:9261dd75-830d-473d-a636-ae6b0bafa5d0}}  swaps the roles of human-generated text, but we still see similar patterns: almost all metrics overrate machines over humans, but the problem is mitigated in COMET-QE, a referenceless, quality estimation metric.
This confirms that our findings hold independently of the design choice.
Fixed-effect coefficients {{formula:6cc62df4-cf15-43c3-b937-d0fe999b3b42}}  from the linear mixed-effects analysis that measures how much automatic metrics overrate machine text over human, as compared to human raters (§REF ). {{formula:11b18b38-85e0-4292-b25c-e18510ec00f8}}  is neutral, and statistical significance is indicated by red (positive) or blue text (negative). The subscripts indicate 90% confidence intervals. magentaCOMET-QE, magentaPrism-src, magentaSummaQA and magentaCLIP-S are referenceless metrics. In both WMT20 ZH-EN and WMT20 EN-DE, Human-B is evaluated as human-generated translations. Human-A (WMT20 ZH-EN) and Human-A and Human-P (WMT20 EN-DE) are used as the reference set for reference-based metrics.
4*ZH-EN
magentaCOMET-QE
Ensemble
COMET
BLEURT
BERTScore
CharacTER
MoverScore
METEOR

{{formula:919449df-4fb3-4986-af10-47868cfc86a4}}
{{formula:c8bed919-f15d-408f-bc5b-803059ba0a52}}
{{formula:3f86d959-9531-4ff8-96ca-fda8562e9d0f}}
{{formula:9f629319-d372-406b-921d-a22f4cf732fe}}
{{formula:e155470f-52b5-442a-8698-b8d0827434be}}
{{formula:18479c7d-8cb1-429e-abb3-376f6a556d29}}
{{formula:144b0a28-5e9d-4b9a-acd5-1bafb23e5e0b}}
{{formula:234efcaa-df01-405f-953f-1a238220aa21}}

Prism-ref
chrF
TER
chrF++
ROUGE-3
BLEU
ROUGE-L
magentaPrism-src

{{formula:69aec8e7-2480-4bf8-92bf-ba869f9ef61d}}
{{formula:fa4718b7-6afb-4b02-b296-932d7fe43c4d}}
{{formula:88db7f6e-57fc-4ecb-a78c-b5610b45e25f}}
{{formula:3d20cd50-fb3c-4efd-97a9-7fce1019ec41}}
{{formula:cc45d1a6-7ab7-4909-a720-f833a3efb36a}}
{{formula:5f5b1017-efde-4c24-a86b-05488800bc04}}
{{formula:882393ae-2ec6-42a6-9e6c-a1efd3ea676a}}
{{formula:b61dcc6c-451c-4647-a253-4b4e5e2d0235}}
4*EN-DE
magentaCOMET-QE
Ensemble
COMET
MoverScore
chrF
chrF++
BLEU
CharacTER

{{formula:a213e893-954f-4fa4-966a-d57030761fa8}}
{{formula:ba6eee5e-5f71-40dc-a1de-096a4c9927db}}
{{formula:e5f1b399-8109-49a9-881e-118839c7cb18}}
{{formula:792a9579-8cbc-42c1-8a6b-f82964d7c827}}
{{formula:4233ee05-a2dd-40aa-9d19-f5c7c841fd13}}
{{formula:380c5f73-7b3e-480a-b351-1b59018df0d0}}
{{formula:abe542ba-b238-41f2-9e7c-d5a6f94be9b9}}
{{formula:c29925a5-6e53-4ca2-992d-bfb0166a891f}}

BERTScore
Prism-ref
TER
magentaPrism-src

{{formula:1460209b-9bbf-4723-96d8-a344c2d79504}}
{{formula:c0a08785-6494-4af1-8b97-76b63dd7cd52}}
{{formula:7e485fb2-2ea1-4968-b9e3-ca2477af1603}}
{{formula:398145d8-d4b0-45c3-a934-3f0ee6272868}}
4*CNNDM
TER
COMET
Ensemble
BERTScore
MoverScore
magentaCOMET-QE
CharacTER
BLEURT

{{formula:d78d580d-b55d-4575-8124-6647a8bfc718}}
{{formula:e9297ee8-ce67-4386-a3fa-4bce53a05b91}}
{{formula:6e2496a2-4895-46f1-bd1d-915ac6f0d4bd}}
{{formula:78adde73-b883-4827-8002-e29a496a5d84}}
{{formula:d86e79c6-4c1c-4496-85dd-8c15338add77}}
{{formula:da701a8b-eeea-4c2d-a85d-4bdf078b9aef}}
{{formula:65b2c4b0-f0c1-4530-9021-2fbff2b23a19}}
{{formula:3b06293b-e44a-477c-9172-1cc90ffe59e0}}

magentaSummaQA
ROUGE-L
BLEU
Prism-ref
chrF
chrF++
ROUGE-3
METEOR

{{formula:4fd64bb6-c16c-465d-a2d9-7c84015c167c}}
{{formula:4be8084c-5eda-485a-abb4-9e050abd9c52}}
{{formula:55939f8d-5dbb-4b8a-b4ad-05b9ccd02749}}
{{formula:0c7ce91b-7073-4bf4-9e39-d6fc3941dab6}}
{{formula:e0807b6a-e1ff-43c8-9952-e79f9f1ec5ab}}
{{formula:f1eee4cc-cc4b-4a8a-be5a-b75764e65346}}
{{formula:9a4d5c20-9b54-4f5a-ac22-bc7a38d4a944}}
{{formula:f3ac7fc5-3de5-48b9-aa49-2f369d777ef5}}
4*COCO
magentaCLIP-S
CharacTER
RefCLIP-S
chrF
SPICE
Ensemble
ROUGE-3
METEOR

{{formula:384646ea-390b-4f78-98b1-206bb7937e38}}
{{formula:876ca066-780d-4469-a6bc-739b82a011cf}}
{{formula:088d7fb1-4f7d-47a3-87a4-bb6e289cc2f5}}
{{formula:3788b41c-b06c-48e3-8268-4879ab4d9340}}
{{formula:ed3048d4-62c7-447a-842e-9aaab05c2649}}
{{formula:22a03f4c-32d2-40f0-ba83-418ddbd05136}}
{{formula:6e9ba137-5a74-46a4-af67-b281b51d6bd0}}
{{formula:a6f075d9-7587-4a1f-a882-5367a51968c0}}

chrF++
RefOnlyC
CIDEr
MoverScore
SentBLEU
TER
ROUGE-L
BERTScore

{{formula:0fbdd0bd-06a4-4550-a87e-3137c54d16e5}}
{{formula:55ca19c4-e4bd-4588-b73a-34ed909f8190}}
{{formula:11a222c6-517e-43b9-8a63-5aa3152866fc}}
{{formula:0a24225c-d92d-4056-a34b-4f0487b99816}}
{{formula:d5b52196-cbd9-42bc-a6be-16137a554d0f}}
{{formula:6afdaa02-62cb-4edd-ab44-90598a1a602f}}
{{formula:523f4011-6b53-4a5f-a7c7-6dff90c37dc9}}
{{formula:0f853571-f554-48b9-89b1-1758b25ebef1}}
{{table:422c2af5-19e6-4c1b-a6c3-61e29c31de6d}}{{table:d7c43f3d-5ce0-4abd-89f6-f52c9a198671}}Fixed-effect coefficients {{formula:c1d1887a-c212-4203-8696-a8704eabb3d2}}  from the linear mixed-effects analysis that measures how much automatic metrics overrate machine text over human, as compared to human raters (§REF ). The roles of human translations are swapped: Human-A is evaluated, and Human-B (WMT20 ZH-EN) and Human-B and Human-P (WMT20 EN-DE) are used as the reference. We still see similar patterns to Table {{table:d7c43f3d-5ce0-4abd-89f6-f52c9a198671}} : almost all automatic metrics overrate machines over humans, but the problem is less severe in the referenceless metric of magentaCOMET-QE.
4*ZH-EN
magentaCOMET-QE
Ensemble
COMET
BLEURT
TER
BERTScore
ROUGE-3
Prism-ref

{{formula:35db3ac5-a2b5-4379-ad59-340f7efe3b7a}}
{{formula:84700b18-54f2-4974-9098-dc02c847729b}}
{{formula:1e7fb113-3fbc-479a-a85a-09ba6a648e8f}}
{{formula:67ccb2b0-a7f6-4cf9-ae78-1481123b9f99}}
{{formula:0b2c56de-44d8-48c0-8ad3-dda4169e23f5}}
{{formula:8d03f387-8594-4bdf-b3ef-12faeac4df1e}}
{{formula:b61105e2-8a30-4874-aec9-7269b296a781}}
{{formula:d2d37624-f830-4ca2-bf1f-0dcec35be955}}

CharacTER
ROUGE-L
chrF
MoverScore
METEOR
chrFpp
BLEU
magentaPrism-src

{{formula:3cdb07d8-e6cf-42b0-b200-d1413fa19a12}}
{{formula:10f4cf53-9b73-455e-9755-d0621d17e771}}
{{formula:c098fb69-7bb5-4baf-bade-e18e2b1674fc}}
{{formula:a6a10d72-4093-4cb6-8947-e5d8f0e9b4e8}}
{{formula:be011534-5051-4c9b-81c3-bd964744f301}}
{{formula:ccd38d63-53f7-4673-8f4a-aecc33e6c970}}
{{formula:9759d746-d267-4bf1-9588-c02ad0752af5}}
{{formula:6d98a10a-3465-4568-bfb9-dbeb5920bdf8}}
4*EN-DE
magentaCOMET-QE
Ensemble
COMET
MoverScore
Prism-ref
chrF
BERTScore
CharacTER

{{formula:7d66f63e-7279-4bf5-ab10-1d6b421433d6}}
{{formula:6e6c23aa-0a14-4308-b1ae-d7be889f99aa}}
-{{formula:81799c97-0fba-4e7c-a17a-dcec4a85a3b7}}
{{formula:e25fff8c-8ae5-47e8-8cdb-d215eb1555a4}}
{{formula:a29df166-655a-4770-b5d1-d4cdf1febc76}}
{{formula:71b07d73-d71e-4056-8c73-a9c28bf630c7}}
{{formula:7a9e83c8-4256-474f-a08d-b28526f27850}}
{{formula:0543cb44-5a0a-4b95-be9b-03e3d5d78964}}

chrF++
SentBLEU
TER
magentaPrism-src

{{formula:d97e3e1d-aac6-4b2c-b3f7-dde533f916fa}}
{{formula:a4b3d82b-6b75-404f-90ca-958b1c973cc8}}
{{formula:5e49bdfa-1c59-4c75-b6fb-618cb075d532}}
{{formula:5c97a8ee-033a-4b47-b977-cb75212a359c}}

{{table:4d37fe13-7c85-41d3-b5b8-3b348c263f55}}{{table:9261dd75-830d-473d-a636-ae6b0bafa5d0}}
Crowdworker vs. Rubric-based Expert Evaluations
Seen in Table {{table:192cd3af-577d-4e06-8fe2-d49ec0b7cc23}}  are examples where crowdworker evaluators {{cite:2ae65e9}} and professional translators {{cite:a7cd7fc}} disagree: crowdworkers give lower scores to the human-generated translations than the machine-generated ones.
The first case requires document-level context to properly evaluate.
Document-level context and diversity in high-quality human translations can mislead crowdworkers.
Examples where crowdworker evaluators {{cite:2ae65e9}} and professional translators {{cite:a7cd7fc}} disagree: crowdworkers give lower scores to the human-generated translations than the machine-generated ones. The first case requires document-level context to properly evaluate. UTF8gbsn兴安省 is Hung Yen Province in Vietnam in this context, but there is entity ambiguity (Xing'an Province that existed in Republic of China.). The second one illustrates the diversity of human generations that mislead crowdworkers.

WMT20 ZH-EN

Source
UTF8gbsn 希望兴安省继续为白俄罗斯企业提供便利条件。
UTF8gbsn 凭的是相机而动的时势驾驭。
Huoshan
It is hoped that Xing'an Province will continue to provide convenient conditions for Belarusian enterprises.
It is based on the current situation of the camera.
Human-A
He hoped that Hung Yen Province would continue to provide convenient conditions for Belarusian enterprises.
This relies on the ability to seize opportunities.
Human-B
He hoped that this could continue in the future.
It is based on the observation of various situations at different times.
{{table:befaeeb5-4c68-414c-b868-7e0e8059b6f2}}{{table:192cd3af-577d-4e06-8fe2-d49ec0b7cc23}}


Soft reward and action dropout for MINERVA

DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning

Abstract: We study the problem of learning to reason in large scale knowledge graphs (KGs). More specifically, we describe a novel reinforcement learning framework for learning multi-hop relational paths: we use a policy-based agent with continuous states based on knowledge graph embeddings, which reasons in a KG vector-space by sampling the most promising relation to extend its path. In contrast to prior work, our approach includes a reward function that takes the accuracy, diversity, and efficiency into consideration. Experimentally, we show that our proposed method outperforms a path-ranking based algorithm and knowledge graph embedding methods on Freebase and Never-Ending Language Learning datasets.

Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning

Abstract: Knowledge bases (KB), both automatically and manually constructed, are often incomplete --- many valid facts can be inferred from the KB by synthesizing existing information. A popular approach to KB completion is to infer new relations by combinatory reasoning over the information found along other paths connecting a pair of entities. Given the enormous size of KBs and the exponential number of paths, previous path-based models have considered only the problem of predicting a missing relation given two entities or evaluating the truth of a proposed triple. Additionally, these methods have traditionally used random paths between fixed entity pairs or more recently learned to pick paths between them. We propose a new algorithm MINERVA, which addresses the much more difficult and practical task of answering questions where the relation is known, but only one entity. Since random walks are impractical in a setting with combinatorially many destinations from a start node, we present a neural reinforcement learning approach which learns how to navigate the graph conditioned on the input query to find predictive paths. Empirically, this approach obtains state-of-the-art results on several datasets, significantly outperforming prior methods.

M-Walk: Learning to Walk over Graphs using Monte Carlo Tree Search

Abstract: Learning to walk over a graph towards a target node for a given query and a source node is an important problem in applications such as knowledge base completion (KBC). It can be formulated as a reinforcement learning (RL) problem with a known state transition model. To overcome the challenge of sparse rewards, we develop a graph-walking agent called M-Walk, which consists of a deep recurrent neural network (RNN) and Monte Carlo Tree Search (MCTS). The RNN encodes the state (i.e., history of the walked path) and maps it separately to a policy and Q-values. In order to effectively train the agent from sparse rewards, we combine MCTS with the neural policy to generate trajectories yielding more positive rewards. From these trajectories, the network is improved in an off-policy manner using Q-learning, which modifies the RNN policy via parameter sharing. Our proposed RL algorithm repeatedly applies this policy-improvement step to learn the model. At test time, MCTS is combined with the neural policy to predict the target node. Experimental results on several graph-walking benchmarks show that M-Walk is able to learn better policies than other RL-based methods, which are mainly based on policy gradients. M-Walk also outperforms traditional KBC baselines.

Multi-Hop Knowledge Graph Reasoning with Reward Shaping

Abstract: Multi-hop reasoning is an effective approach for query answering (QA) over incomplete knowledge graphs (KGs). The problem can be formulated in a reinforcement learning (RL) setup, where a policy-based agent sequentially extends its inference path until it reaches a target. However, in an incomplete KG environment, the agent receives low-quality rewards corrupted by false negatives in the training data, which harms generalization at test time. Furthermore, since no golden action sequence is used for training, the agent can be misled by spurious search trajectories that incidentally lead to the correct answer. We propose two modeling advances to address both issues: (1) we reduce the impact of false negative supervision by adopting a pretrained one-hop embedding model to estimate the reward of unobserved facts; (2) we counter the sensitivity to spurious paths of on-policy RL by forcing the agent to explore a diverse set of paths using randomly generated edge masks. Our approach significantly improves over existing path-based KGQA models on several benchmark datasets and is comparable or better than embedding-based models.

Reinforcement Knowledge Graph Reasoning for Explainable Recommendation

Abstract: Recent advances in personalized recommendation have sparked great interest in the exploitation of rich structured information provided by knowledge graphs. Unlike most existing approaches that only focus on leveraging knowledge graphs for more accurate recommendation, we aim to conduct explicit reasoning with knowledge for decision making so that the recommendations are generated and supported by an interpretable causal inference procedure. To this end, we propose a method called Policy-Guided Path Reasoning (PGPR), which couples recommendation and interpretability by providing actual paths in a knowledge graph. Our contributions include four aspects. We first highlight the significance of incorporating knowledge graphs into recommendation to formally define and interpret the reasoning process. Second, we propose a reinforcement learning (RL) approach featured by an innovative soft reward strategy, user-conditional action pruning and a multi-hop scoring function. Third, we design a policy-guided graph search algorithm to efficiently and effectively sample reasoning paths for recommendation. Finally, we extensively evaluate our method on several large-scale real-world benchmark datasets, obtaining favorable results compared with state-of-the-art methods.

Learning Collaborative Agents with Rule Guidance for Knowledge Graph Reasoning

Abstract: Walk-based models have shown their unique advantages in knowledge graph (KG) reasoning by achieving state-of-the-art performance while allowing for explicit visualization of the decision sequence. However, the sparse reward signals offered by the KG during a traversal are often insufficient to guide a sophisticated reinforcement learning (RL) model. An alternate approach to KG reasoning is using traditional symbolic methods (e.g., rule induction), which achieve high precision without learning but are hard to generalize due to the limitation of symbolic representation. In this paper, we propose to fuse these two paradigms to get the best of both worlds. Our method leverages high-quality rules generated by symbolic-based methods to provide reward supervision for walk-based agents. Due to the structure of symbolic rules with their entity variables, we can separate our walk-based agent into two sub-agents thus allowing for additional efficiency. Experiments on public datasets demonstrate that walk-based models can benefit from rule guidance significantly.

Reasoning on Knowledge Graphs with Debate Dynamics

Abstract: We propose a novel method for automatic reasoning on knowledge graphs based on debate dynamics. The main idea is to frame the task of triple classification as a debate game between two reinforcement learning agents which extract arguments -- paths in the knowledge graph -- with the goal to promote the fact being true (thesis) or the fact being false (antithesis), respectively. Based on these arguments, a binary classifier, called the judge, decides whether the fact is true or false. The two agents can be considered as sparse, adversarial feature generators that present interpretable evidence for either the thesis or the antithesis. In contrast to other black-box methods, the arguments allow users to get an understanding of the decision of the judge. Since the focus of this work is to create an explainable method that maintains a competitive predictive accuracy, we benchmark our method on the triple classification and link prediction task. Thereby, we find that our method outperforms several baselines on the benchmark datasets FB15k-237, WN18RR, and Hetionet. We also conduct a survey and find that the extracted arguments are informative for users.

Sample of RL reasoning algorithms

Most of the approaches use path based RL reasoning, which sequentially traverses triples in a graph to make predictions. Because of this, it provides transparent reasoning for every prediction. An issue with this approach can be convergence and generating relevant reasoning paths. To help, demonstrations of reasoning can be provided by metapaths which acts as a form of weak supervision {{cite:bcd3869}}, {{cite:f236136}}. RL reasoning applied to the standard benchmarks graphs used in research has not shown promising performance and thus has not been the focus of much research. A small sample of path-based models and their characteristics are shown in Table {{table:3274ec56-9e31-4271-88a5-49a77a4999ec}} .


CEEMD, ARIMA, and sparse Bayesian learning

Ridge regression, LASSO, support vector regression (SVR), BPNN, and random forest (RF)

A two-layer NMF (non-negative matrix factorization) model and GIHS (giant information history simulation)

Using four different online media sources to forecast the crude oil price

Abstract: This study looks for signals of economic awareness on online social media and tests their significance in economic predictions. The study analyses, over a period of two years, the relationship between the West Texas Intermediate daily crude oil price and multiple predictors extracted from Twitter, Google Trends, Wikipedia, and the Global Data on Events, Language, and Tone database (GDELT). Semantic analysis is applied to study the sentiment, emotionality and complexity of the language used. Autoregressive Integrated Moving Average with Explanatory Variable (ARIMAX) models are used to make predictions and to confirm the value of the study variables. Results show that the combined analysis of the four media platforms carries valuable information in making financial forecasting. Twitter language complexity, GDELT number of articles and Wikipedia page reads have the highest predictive power. This study also allows a comparison of the different fore-sighting abilities of each platform, in terms of how many days ahead a platform can predict a price movement before it happens. In comparison with previous work, more media sources and more dimensions of the interaction and of the language used are combined in a joint analysis.

An Adaptive Hybrid Learning Paradigm Integrating CEEMD, ARIMA and SBL for Crude Oil Price Forecasting

Abstract: Crude oil is one of the main energy sources and its prices have gained increasing attention due to its important role in the world economy. Accurate prediction of crude oil prices is an important issue not only for ordinary investors, but also for the whole society. To achieve the accurate prediction of nonstationary and nonlinear crude oil price time series, an adaptive hybrid ensemble learning paradigm integrating complementary ensemble empirical mode decomposition (CEEMD), autoregressive integrated moving average (ARIMA) and sparse Bayesian learning (SBL), namely CEEMD-ARIMA&SBL-SBL (CEEMD-A&S-SBL), is developed in this study. Firstly, the decomposition method CEEMD, which can reduce the end effects and mode mixing, was employed to decompose the original crude oil price time series into intrinsic mode functions (IMFs) and one residue. Then, ARIMA and SBL with combined kernels were applied to predict target values for the residue and each single IMF independently. Finally, the predicted values of the above two models for each component were adaptively selected based on the training precision, and then aggregated as the final forecasting results using SBL without kernel-tricks. Experiments were conducted on the crude oil spot prices of the West Texas Intermediate (WTI) and Brent crude oil to evaluate the performance of the proposed CEEMD-A&S-SBL. The experimental results demonstrated that, compared with some state-of-the-art prediction models, CEEMD-A&S-SBL can significantly improve the prediction accuracy of crude oil prices in terms of the root mean squared error (RMSE), the mean absolute percent error (MAPE), and the directional statistic (Dstat).

Forecasting Oil Price Using Web-based Sentiment Analysis

Abstract: International oil price forecasting is a complex and important issue in the research area of energy economy. In this paper, a new model based on web-based sentiment analysis is proposed. For the oil market, sentiment analysis is used to extract key information from web texts from the four perspectives of: compound, negative, neutral, and positive sentiment. These are constructed as feature and input into oil price forecasting models with oil price itself. Finally, we analyze the effect in various views and get some interesting discoveries. The results show that the root mean squared error can be reduced by about 0.2 and the error variance by 0.2, which means that the accuracy and stability are thereby improved. Furthermore, we find that different types of sentiments can all improve performance but by similar amounts. Last but not least, text with strong intensity can better support oil price forecasting than weaker text, for which the root mean squared error can be reduced by up to 0.5, and the number of the bad cases is reduced by 20%, indicating that text with strong intensity can correct the original oil price forecast. We believe that our research will play a strong supporting role in future research on using web information for oil price forecasting.

Forecasting Oil Price Volatility in the Era of Big Data: A Text Mining for VaR Approach

Abstract: The rapid fluctuations in global crude oil prices are one of the important factors affecting both the sustainable development and the green transformation of the global economy. To accurately measure the risks of crude oil prices, in the context of big data, this study introduces the two-layer non-negative matrix factorization model, a kind of natural language processing, to extract the dynamic risk factors from online news and assign them as weighted factors to historical data. Finally, this study proposes a giant information history simulation (GIHS) method which is used to forecast the value-at-risk (VaR) of crude oil. In conclusion, this paper shows that considering the impact of dynamic risk factors from online news on the VaR can improve the accuracy of crude oil VaR measurement, providing an effective tool for analyzing crude oil price risks in oil market, providing risk management support for international oil market investors, and providing the country with a sense of risk analysis to achieve sustainable and green transformation.

Summary of recent studies on crude oil price forecasting.

This section reviews existing typical literature on crude oil price forecasting and summarizes the literature in Table {{table:77392b3a-cdb3-426f-bfdc-1ac206935487}} . More details about the leading technologies used in the proposed framework AGESL are followed.


Retinoic Acid Induces Differentiation of Mouse F9 Embryonic Carcinoma Cell by Modulating the miR-485 Targeting of Abhd2

Abstract: Retinoic acid (RA) plays a key role in pluripotent cell differentiation. In F9 embryonic carcinoma cells, RA can induce differentiation towards somatic lineages via the Ras-extracellular signal-regulated kinase (Ras/Erk) pathway, but the mechanism through which it induces the Erk1/2 phosphorylation is unclear. Here, we show that miR-485 is a positive regulator that targets α/β-hydrolase domain-containing protein 2 (Abhd2), which can result in Erk1/2 phosphorylation and triggers differentiation. RA up-regulates miR-485 and concurrently down-regulates Abhd2. We verified that Abhd2 is targeted by miR-485 and they both can influence the phosphorylation of Erk1/2. In summary, RA can mediate cell differentiation by phosphorylating Erk1/2 via miR-485 and Abhd2.

Discovery of cancer common and specific driver gene sets

Abstract: Cancer is known as a disease mainly caused by gene alterations. Discovery of mutated driver pathways or gene sets is becoming an important step to understand molecular mechanisms of carcinogenesis. However, systematically investigating commonalities and specificities of driver gene sets among multiple cancer types is still a great challenge, but this investigation will undoubtedly benefit deciphering cancers and will be helpful for personalized therapy and precision medicine in cancer treatment. In this study, we propose two optimization models to \emph{de novo} discover common driver gene sets among multiple cancer types (ComMDP) and specific driver gene sets of one certain or multiple cancer types to other cancers (SpeMDP), respectively. We first apply ComMDP and SpeMDP to simulated data to validate their efficiency. Then, we further apply these methods to 12 cancer types from The Cancer Genome Atlas (TCGA) and obtain several biologically meaningful driver pathways. As examples, we construct a common cancer pathway model for BRCA and OV, infer a complex driver pathway model for BRCA carcinogenesis based on common driver gene sets of BRCA with eight cancer types, and investigate specific driver pathways of the liquid cancer lymphoblastic acute myeloid leukemia (LAML) versus other solid cancer types. In these processes more candidate cancer genes are also found.

Reversing the Intractable Nature of Pancreatic Cancer by Selectively Targeting ALDH-High, Therapy-Resistant Cancer Cells

Abstract: Human pancreatic ductal adenocarcinoma (PDAC) is a cancer with a dismal prognosis. The efficacy of PDAC anticancer therapies is often short-lived; however, there is little information on how this disease entity so frequently gains resistance to treatment. We adopted the concept of cancer stem cells (CSCs) to explain the mechanism of resistance and evaluated the efficacy of a candidate anticancer drug to target these therapy-resistant CSCs. We identified a subpopulation of cells in PDAC with CSC features that were enriched for aldehyde dehydrogenase (ALDH), a marker expressed in certain stem/progenitor cells. These cells were also highly resistant to, and were further enriched by, treatment with gemcitabine. Similarly, surgical specimens from PDAC patients showed that those who had undergone preoperative chemo-radiation therapy more frequently displayed cancers with ALDH strongly positive subpopulations compared with untreated patients. Importantly, these ALDH-high cancer cells were sensitive to disulfiram, an ALDH inhibitor, when tested in vitro. Furthermore, in vivo xenograft studies showed that the effect of disulfiram was additive to that of low-dose gemcitabine when applied in combination. In conclusion, human PDAC-derived cells that express high levels of ALDH show CSC features and have a key role in the development of resistance to anticancer therapies. Disulfiram can be used to suppress this therapy-resistant subpopulation.

Nineteen genes selected as features from the literature.

Phase 1: Applying Machine Learning Algorithms

Through literature, it was found that nineteen genes play a significant role in monitoring actin production and acinar and ductal cell function, and tend to be overexpressed in PC.
Thus, they were selected as features for the five classifiers.
The nineteen genes are shown in Table {{table:001d36ef-25d6-4cc9-8f48-c3c8eb9ce575}} .


Cross-Modality Relevance for Reasoning on Language and Vision

Abstract: This work deals with the challenge of learning and reasoning over language and vision data for the related downstream tasks such as visual question answering (VQA) and natural language for visual reasoning (NLVR). We design a novel cross-modality relevance module that is used in an end-to-end framework to learn the relevance representation between components of various input modalities under the supervision of a target task, which is more generalizable to unobserved data compared to merely reshaping the original representation space. In addition to modeling the relevance between the textual entities and visual entities, we model the higher-order relevance between entity relations in the text and object relations in the image. Our proposed approach shows competitive performance on two different language and vision tasks using public benchmarks and improves the state-of-the-art published results. The learned alignments of input spaces and their relevance representations by NLVR task boost the training efficiency of VQA task.

Heterogeneous Graph Learning for Visual Commonsense Reasoning

Abstract: Visual commonsense reasoning task aims at leading the research field into solving cognition-level reasoning with the ability to predict correct answers and meanwhile providing convincing reasoning paths, resulting in three sub-tasks i.e., Q->A, QA->R and Q->AR. It poses great challenges over the proper semantic alignment between vision and linguistic domains and knowledge reasoning to generate persuasive reasoning paths. Existing works either resort to a powerful end-to-end network that cannot produce interpretable reasoning paths or solely explore intra-relationship of visual objects (homogeneous graph) while ignoring the cross-domain semantic alignment among visual concepts and linguistic words. In this paper, we propose a new Heterogeneous Graph Learning (HGL) framework for seamlessly integrating the intra-graph and inter-graph reasoning in order to bridge the vision and language domain. Our HGL consists of a primal vision-to-answer heterogeneous graph (VAHG) module and a dual question-to-answer heterogeneous graph (QAHG) module to interactively refine reasoning paths for semantic agreement. Moreover, our HGL integrates a contextual voting module to exploit a long-range visual context for better global reasoning. Experiments on the large-scale Visual Commonsense Reasoning benchmark demonstrate the superior performance of our proposed modules on three tasks (improving 5% accuracy on Q->A, 3.5% on QA->R, 5.8% on Q->AR).

From Recognition to Cognition: Visual Commonsense Reasoning

Abstract: Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA problems derived from 110k movie scenes. The key recipe for generating non-trivial and high-quality problems at scale is Adversarial Matching, a new approach to transform rich annotations into multiple choice questions with minimal bias. Experimental results show that while humans find VCR easy (over 90% accuracy), state-of-the-art vision models struggle (~45%). To move towards cognition-level understanding, we present a new reasoning engine, Recognition to Cognition Networks (R2C), that models the necessary layered inferences for grounding, contextualization, and reasoning. R2C helps narrow the gap between humans and machines (~65%); still, the challenge is far from solved, and we provide analysis that suggests avenues for future work.

Latest research in Visual Commonsense Reasoning. CTM: Contextual Voting Module, CC: Conceptual Captions, KV: Key Value

The growing focus on reasoning based systems have not only led to instigation of various relevant datasets but also popularized the idea of visual reasoning (see Table {{table:2e95048a-59aa-4eca-a256-0484506b6d6c}} ). VCR  {{cite:4ecf3a8}}, often viewed through the lens of an extension of the VQA task, is slowly and steadily gaining extensive popularity in recent literature owing to its contribution in designing socially relevant interpretable models. Most VCR systems have focused on utilizing embeddings generated from pre-trained transformer networks (detailed in Section REF ) to obtain high-quality latent representations for visual and language modalities. This evolution has also prompted casuality, counterfactual inference and contrastive learning to flourish in these domains of interest.


TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

Abstract: Medical image segmentation is an essential prerequisite for developing healthcare systems, especially for disease diagnosis and treatment planning. On various medical image segmentation tasks, the u-shaped architecture, also known as U-Net, has become the de-facto standard and achieved tremendous success. However, due to the intrinsic locality of convolution operations, U-Net generally demonstrates limitations in explicitly modeling long-range dependency. Transformers, designed for sequence-to-sequence prediction, have emerged as alternative architectures with innate global self-attention mechanisms, but can result in limited localization abilities due to insufficient low-level details. In this paper, we propose TransUNet, which merits both Transformers and U-Net, as a strong alternative for medical image segmentation. On one hand, the Transformer encodes tokenized image patches from a convolution neural network (CNN) feature map as the input sequence for extracting global contexts. On the other hand, the decoder upsamples the encoded features which are then combined with the high-resolution CNN feature maps to enable precise localization. We argue that Transformers can serve as strong encoders for medical image segmentation tasks, with the combination of U-Net to enhance finer details by recovering localized spatial information. TransUNet achieves superior performances to various competing methods on different medical applications including multi-organ segmentation and cardiac segmentation. Code and models are available at https://github.com/Beckschen/TransUNet.

UNetFormer: A Unified Vision Transformer Model and Pre-Training Framework for 3D Medical Image Segmentation

Abstract: Vision Transformers (ViT)s have recently become popular due to their outstanding modeling capabilities, in particular for capturing long-range information, and scalability to dataset and model sizes which has led to state-of-the-art performance in various computer vision and medical image analysis tasks. In this work, we introduce a unified framework consisting of two architectures, dubbed UNetFormer, with a 3D Swin Transformer-based encoder and Convolutional Neural Network (CNN) and transformer-based decoders. In the proposed model, the encoder is linked to the decoder via skip connections at five different resolutions with deep supervision. The design of proposed architecture allows for meeting a wide range of trade-off requirements between accuracy and computational cost. In addition, we present a methodology for self-supervised pre-training of the encoder backbone via learning to predict randomly masked volumetric tokens using contextual information of visible tokens. We pre-train our framework on a cohort of $5050$ CT images, gathered from publicly available CT datasets, and present a systematic investigation of various components such as masking ratio and patch size that affect the representation learning capability and performance of downstream tasks. We validate the effectiveness of our pre-training approach by fine-tuning and testing our model on liver and liver tumor segmentation task using the Medical Segmentation Decathlon (MSD) dataset and achieve state-of-the-art performance in terms of various segmentation metrics. To demonstrate its generalizability, we train and test the model on BraTS 21 dataset for brain tumor segmentation using MRI images and outperform other methods in terms of Dice score. Code: https://github.com/Project-MONAI/research-contributions

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

Abstract: Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image. One of the key challenges behind this task is leveraging the referring expression for highlighting relevant positions in the image. A paradigm for tackling this problem is to leverage a powerful vision-language (“cross-madal”) decoder to fuse features independently extracted from a vision encoder and a language encoder. Recent methods have made remarkable advancements in this paradigm by exploiting Transformers as cross-modal decoders, concurrent to the Transformer's overwhelming success in many other vision-language tasks. Adopting a different approach in this work, we show that significantly better cross-modal alignments can be achieved through the early fusion of linguistic and visual features in intermediate layers of a vision Transformer encoder network. By conducting cross-modal feature fusion in the visual feature encoding stage, we can leverage the well-proven correlation modeling power of a Transformer encoder for excavating helpful multi-modal context. This way, accurate segmentation results are readily harvested with a light-weight mask predictor. Without bells and whistles, our method surpasses the previous state-of-the-art methods on Ref CoCo, RefCOCO+, and G-Ref by large margins.

ViT for Image Segmentation.

Table {{table:5ad48c8f-c361-49dd-af93-452f8eb5c93d}}  shows the application of ViT for the image segmentation.


Data quality is hard to test, and may have unexpected consequences.

ML testing goes beyond programming bugs to issues that arise from model, data errors, or uncertainty.

Design separate modules/services for data quality assessment.

Software Engineering Practice in the Development of Deep Learning Applications

Abstract: Deep-Learning(DL) applications have been widely employed to assist in various tasks. They are constructed based on a data-driven programming paradigm that is different from conventional software applications. Given the increasing popularity and importance of DL applications, software engineering practitioners have some techniques specifically for them. However, little research is conducted to identify the challenges and lacks in practice. To fill this gap, in this paper, we surveyed 195 practitioners to understand their insight and experience in the software engineering practice of DL applications. Specifically, we asked the respondents to identify lacks and challenges in the practice of the development life cycle of DL applications. The results present 13 findings that provide us with a better understanding of software engineering practice of DL applications. Further, we distil these findings into 7 actionable recommendations for software engineering researchers and practitioners to improve the development of DL applications.

Software Engineering Challenges of Deep Learning

Abstract: Surprisingly promising results have been achieved by deep learning (DL) systems in recent years. Many of these achievements have been reached in academic settings, or by large technology companies with highly skilled research groups and advanced supporting infrastructure. For companies without large research groups or advanced infrastructure, building high-quality production-ready systems with DL components has proven challenging. There is a clear lack of well-functioning tools and best practices for building DL systems. It is the goal of this research to identify what the main challenges are, by applying an interpretive research approach in close collaboration with companies of varying size and type. A set of seven projects have been selected to describe the potential with this new technology and to identify associated main challenges. A set of 12 main challenges has been identified and categorized into the three areas of development, production, and organizational challenges. Furthermore, a mapping between the challenges and the projects is defined, together with selected motivating descriptions of how and why the challenges apply to specific projects. Compared to other areas such as software engineering or database technologies, it is clear that DL is still rather immature and in need of further work to facilitate development of high-quality systems. The challenges identified in this paper can be used to guide future research by the software engineering and DL communities. Together, we could enable a large number of companies to start taking advantage of the high potential of the DL technology.

List of SA challenges for ML and related solutions as extracted from the SLR.

See Table {{table:6b561ca3-1757-4738-8b60-2d07246ffe59}} .


new anomaly detection method (brain mestastases)

DistGP-Seg. Incooperating DistGP into CNN

(Wasserstein based) MaDGAN using self attention (paired)

Unsupervised brain lesion segmentation from MRI using a convolutional autoencoder

Abstract: Lesions that appear hyperintense in both Fluid Attenuated Inversion Recovery (FLAIR) and T2-weighted magnetic resonance images (MRIs) of the human brain are common in the brains of the elderly population and may be caused by ischemia or demyelination. Lesions are biomarkers for various neurodegenerative diseases, making accurate quantification of them important for both disease diagnosis and progression. Automatic lesion detection using supervised learning requires manually annotated images, which can often be impractical to acquire. Unsupervised lesion detection, on the other hand, does not require any manual delineation; however, these methods can be challenging to construct due to the variability in lesion load, placement of lesions, and voxel intensities. Here we present a novel approach to address this problem using a convolutional autoencoder, which learns to segment brain lesions as well as the white matter, gray matter, and cerebrospinal fluid by reconstructing FLAIR images as conical combinations of softmax layer outputs generated from the corresponding T1, T2, and FLAIR images. Some of the advantages of this model are that it accurately learns to segment lesions regardless of lesion load, and it can be used to quickly and robustly segment new images that were not in the training set. Comparisons with state-of-the-art segmentation methods evaluated on ground truth manual labels indicate that the proposed method works well for generating accurate lesion segmentations without the need for manual annotations.

Distributional Gaussian Process Layers for Outlier Detection in Image Segmentation

Abstract: We propose a parameter efficient Bayesian layer for hierarchical convolutional Gaussian Processes that incorporates Gaussian Processes operating in Wasserstein-2 space to reliably propagate uncertainty. This directly replaces convolving Gaussian Processes with a distance-preserving affine operator on distributions. Our experiments on brain tissue-segmentation show that the resulting architecture approaches the performance of well-established deterministic segmentation algorithms (U-Net), which has never been achieved with previous hierarchical Gaussian Processes. Moreover, by applying the same segmentation model to out-of-distribution data (i.e., images with pathology such as brain tumors), we show that our uncertainty estimates result in out-of-distribution detection that outperforms the capabilities of previous Bayesian networks and reconstruction-based approaches that learn normative distributions.

Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images

Abstract: Reliably modeling normality and differentiating abnormal appearances from normal cases is a very appealing approach for detecting pathologies in medical images. A plethora of such unsupervised anomaly detection approaches has been made in the medical domain, based on statistical methods, content-based retrieval, clustering and recently also deep learning. Previous approaches towards deep unsupervised anomaly detection model patches of normal anatomy with variants of Autoencoders or GANs, and detect anomalies either as outliers in the learned feature space or from large reconstruction errors. In contrast to these patch-based approaches, we show that deep spatial autoencoding models can be efficiently used to capture normal anatomical variability of entire 2D brain MR images. A variety of experiments on real MR data containing MS lesions corroborates our hypothesis that we can detect and even delineate anomalies in brain MR images by simply comparing input images to their reconstruction. Results show that constraints on the latent space and adversarial training can further improve the segmentation performance over standard deep representation learning.

MADGAN: unsupervised Medical Anomaly Detection GAN using multiple adjacent brain MRI slice reconstruction

Abstract: Unsupervised learning can discover various unseen abnormalities, relying on large-scale unannotated medical images of healthy subjects. Towards this, unsupervised methods reconstruct a 2D/3D single medical image to detect outliers either in the learned feature space or from high reconstruction loss. However, without considering continuity between multiple adjacent slices, they cannot directly discriminate diseases composed of the accumulation of subtle anatomical anomalies, such as Alzheimer's Disease (AD). Moreover, no study has shown how unsupervised anomaly detection is associated with either disease stages, various (i.e., more than two types of) diseases, or multi-sequence Magnetic Resonance Imaging (MRI) scans. Therefore, we propose unsupervised Medical Anomaly Detection Generative Adversarial Network (MADGAN), a novel two-step method using GAN-based multiple adjacent brain MRI slice reconstruction to detect brain anomalies at different stages on multi-sequence structural MRI: (Reconstruction) Wasserstein loss with Gradient Penalty + 100 L1 loss-trained on 3 healthy brain axial MRI slices to reconstruct the next 3 ones-reconstructs unseen healthy/abnormal scans; (Diagnosis) Average L2 loss per scan discriminates them, comparing the ground truth/reconstructed slices. For training, we use two different datasets composed of 1,133 healthy T1-weighted (T1) and 135 healthy contrast-enhanced T1 (T1c) brain MRI scans for detecting AD and brain metastases/various diseases, respectively. Our Self-Attention MADGAN can detect AD on T1 scans at a very early stage, Mild Cognitive Impairment (MCI), with Area Under the Curve (AUC) 0.727, and AD at a late stage with AUC 0.894, while detecting brain metastases on T1c scans with AUC 0.921.

Unsupervised Lesion Detection via Image Restoration with a Normative Prior

Abstract: Unsupervised lesion detection is a challenging problem that requires accurately estimating normative distributions of healthy anatomy and detecting lesions as outliers without training examples. Recently, this problem has received increased attention from the research community following the advances in unsupervised learning with deep learning. Such advances allow the estimation of high-dimensional distributions, such as normative distributions, with higher accuracy than previous methods.The main approach of the recently proposed methods is to learn a latent-variable model parameterized with networks to approximate the normative distribution using example images showing healthy anatomy, perform prior-projection, i.e. reconstruct the image with lesions using the latent-variable model, and determine lesions based on the differences between the reconstructed and original images. While being promising, the prior-projection step often leads to a large number of false positives. In this work, we approach unsupervised lesion detection as an image restoration problem and propose a probabilistic model that uses a network-based prior as the normative distribution and detect lesions pixel-wise using MAP estimation. The probabilistic model punishes large deviations between restored and original images, reducing false positives in pixel-wise detections. Experiments with gliomas and stroke lesions in brain MRI using publicly available datasets show that the proposed approach outperforms the state-of-the-art unsupervised methods by a substantial margin, +0.13 (AUC), for both glioma and stroke detection. Extensive model analysis confirms the effectiveness of MAP-based image restoration.

Table consisting of fMRI image based results of the brain obtained by the literature review

The semi-exhaustive literature review resulted in 45 research items, from which further 6 were removed due to not containing applications in medical imaging (only exemplar stated in abstract) or being non-available. The resulting papers were further clustered into 5 categories (corresponding to Tab. {{table:586d8cd2-d882-4536-a90e-d9e6b792e45b}} -{{table:6b585252-d90d-46db-94b0-1b1db14ac7f6}}  by their imaging methods. Tab. {{table:586d8cd2-d882-4536-a90e-d9e6b792e45b}}  shows papers applying anomaly detection to occular medical images with retinal fundus images and optical coherence tomography (OCT). Tab. {{table:2f8ce732-b2a8-4848-afc9-5dff2b3298fa}}  focuses on papers with applications in the center body region, with chest X-rays and mammography. Tab. {{table:758a0f69-7649-482d-bc91-f5a73ad646b1}}  summarizes application papers, using CT and functional magnetic resonance imaging (fMRI). Tab. {{table:fac491b7-86b6-4824-beb9-d4b963abc220}}  displays papers applying ML to brain Magnetic resonance imaging (MRI) datasets. Tab. {{table:6b585252-d90d-46db-94b0-1b1db14ac7f6}}  shows mixed applications from the domains of breast ultrasound, chest radiographs, histology and fundus images as well as multi-spectral imaging (MSI).