Training Verifiers to Solve Math Word Problems

Abstract: State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.

PAL: Program-aided Language Models

Abstract: Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks, when provided with a few examples at test time ("few-shot prompting"). Much of this success can be attributed to prompting methods such as "chain-of-thought'', which employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LLMs seem to be adept at this sort of step-by-step decomposition, LLMs often make logical and arithmetic mistakes in the solution part, even when the problem is decomposed correctly. In this paper, we present Program-Aided Language models (PAL): a novel approach that uses the LLM to read natural language problems and generate programs as the intermediate reasoning steps, but offloads the solution step to a runtime such as a Python interpreter. With PAL, decomposing the natural language problem into runnable steps remains the only learning task for the LLM, while solving is delegated to the interpreter. We demonstrate this synergy between a neural LLM and a symbolic interpreter across 13 mathematical, symbolic, and algorithmic reasoning tasks from BIG-Bench Hard and other benchmarks. In all these natural language reasoning tasks, generating code using an LLM and reasoning using a Python interpreter leads to more accurate results than much larger models. For example, PAL using Codex achieves state-of-the-art few-shot accuracy on the GSM8K benchmark of math word problems, surpassing PaLM-540B which uses chain-of-thought by absolute 15% top-1. Our code and data are publicly available at http://reasonwithpal.com/ .

Are NLP Models really able to Solve Simple Math Word Problems?

Abstract: The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered "solved" with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.

Solving General Arithmetic Word Problems

Abstract: This paper presents a novel approach to automatically solving arithmetic word problems. This is the first algorithmic approach that can handle arithmetic problems with multiple steps and operations, without depending on additional annotations or predefined templates. We develop a theory for expression trees that can be used to represent and evaluate the target arithmetic expressions; we use it to uniquely decompose the target arithmetic problem to multiple classification problems; we then compose an expression tree, combining these with world knowledge through a constrained inference framework. Our classifiers gain from the use of {\em quantity schemas} that supports better extraction of features. Experimental results show that our method outperforms existing systems, achieving state of the art performance on benchmark datasets of arithmetic word problems.

We introduce Corex in this paper,
a suite of strategies that transform LLMs into autonomous agents,
thereby leveraging multi-model collaboration for complex reasoning.
This offers a preliminary exploration into the LLM-based multi-model ecosystems.
Through unlocking the synergies among LLMs,
Corex empowers reasoning with enhanced factuality,
faithfulness,
and reliability through various collaboration paradigms.
We conduct extensive evaluations across 18 tasks within 4 categories,
and the results demonstrate superior performance compared to previous solutions.
Moreover,
our methods also exhibit multiple notable advantages including being task-agnostic,
cost-effective,
and annotation-efficient.
We hope that this work may serve as a foundation for further research,
offering novel perspectives in complex reasoning,
collective intelligence,
and autonomous agents.
Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le.
Towards a human-like open-domain chatbot.
CoRR, 2020.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.
Training a helpful and harmless assistant with reinforcement learning from human feedback.
ArXiv preprint, 2022.
BIG bench authors.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
Transactions on Machine Learning Research, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners.
In NeurIPS, 2020.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu.
Chateval: Towards better llm-based evaluators through multi-agent debate, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al.
Evaluating large language models trained on code.
arXiv preprint arXiv:2107.03374, 2021a.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen.
Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2022a.
Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu.
Exploring the use of large language models for reference-free text quality evaluation: An empirical study, 2023.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang.
FinQA: A dataset of numerical reasoning over financial data.
In Proc. of EMNLP, 2021b.
Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang.
ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering.
In Proc. of EMNLP, 2022b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.
Palm: Scaling language modeling with pathways, 2022.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.
BoolQ: Exploring the surprising difficulty of natural yes/no questions.
In Proc. of NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
ArXiv, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems, 2021.
Ali Dorri, Salil S. Kanhere, and Raja Jurdak.
Multi-agent systems: A survey.
IEEE Access, 2018.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch.
Improving factuality and reasoning in language models through multiagent debate.
arXiv preprint arXiv:2305.14325, 2023.
Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot.
Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance, 2023a.
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata.
Improving language model negotiation with self-play and in-context learning from ai feedback, 2023b.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot.
Complexity-based prompting for multi-step reasoning.
In Proc. of ICLR, 2023c.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.
Pal: Program-aided language models.
arXiv preprint arXiv:2211.10435, 2022.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen.
Enabling large language models to generate text with citations, 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant.
Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.
Transactions of the Association for Computational Linguistics, 2021.
Hangfeng He, Hongming Zhang, and Dan Roth.
Rethinking with retrieval: Faithful large language model inference, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt.
Aligning AI with shared human values.
In Proc. of ICLR, 2021.
Marius Hessenthaler, Emma Strubell, Dirk Hovy, and Anne Lauscher.
Bridging fairness and environmental sustainability in natural language processing.
In Proc. of EMNLP, 2022.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman.
Learning to solve arithmetic word problems with verb categorization.
In Proc. of EMNLP, 2014.
Jie Huang and Kevin Chen-Chuan Chang.
Towards reasoning in large language models: A survey.
In Proc. of ACL Findings, 2023.
Alon Jacovi and Yoav Goldberg.
Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?
In Proc. of ACL, 2020.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung.
Survey of hallucination in natural language generation.
ACM Comput. Surv., 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models, 2020.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal.
Decomposed prompting: A modular approach for solving complex tasks.
In Proc. of ICLR, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
Large language models are zero-shot reasoners.
In Proc. of NeurIPS, 2022.
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi.
MAWPS: A math word problem repository.
In Proc. of NAACL, 2016.
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez.
Measuring faithfulness in chain-of-thought reasoning, 2023.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.
Camel: Communicative agents for "mind" exploration of large scale language model society, 2023a.
Jiwei Li and Dan Jurafsky.
Mutual information and diverse decoding improve neural machine translation.
ArXiv, 2016.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis.
Contrastive decoding: Open-ended text generation as optimization.
In Proc. of ACL, 2023b.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen.
Making language models better reasoners with step-aware verifier.
In Proc. of ACL, 2023c.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi.
Encouraging divergent thinking in large language models through multi-agent debate.
arXiv preprint arXiv:2305.19118, 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom.
Program induction by rationale generation: Learning to solve and explain algebraic word problems.
In Proc. of ACL, 2017.
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M. Dai.
Mind's eye: Grounded language model reasoning through simulation.
In Proc. of ICLR, 2023a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang.
Agentbench: Evaluating llms as agents.
arXiv preprint arXiv: 2308.03688, 2023b.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao.
Chameleon: Plug-and-play compositional reasoning with large language models.
arXiv preprint arXiv:2304.09842, 2023a.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan.
Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.
In Proc. of ICLR, 2023b.
Andrea I Luppi, Pedro AM Mediano, Fernando E Rosas, Negin Holland, Tim D Fryer, John T O’Brien, James B Rowe, David K Menon, Daniel Bor, and Emmanuel A Stamatakis.
A synergistic core for human brain evolution and cognition.
Nature Neuroscience, 2022.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch.
Faithful chain-of-thought reasoning.
arXiv preprint arXiv:2301.13379, 2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.
Can a suit of armor conduct electricity? a new dataset for open book question answering.
In Proc. of EMNLP, 2018.
Marvin Minsky.
Society of mind.
Simon and Schuster, 1988.
Marvin Minsky.
The emotion machine: Commonsense thinking, artificial intelligence, and the future of the human mind.
Simon and Schuster, 2007.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman.
Webgpt: Browser-assisted question-answering with human feedback, 2022.
Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin.
Lever: Learning to verify language-to-code generation with execution.
In Proc. of ICML, 2023.
Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama.
Demystifying gpt self-repair for code generation, 2023.
OpenAI.
GPT-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback.
In Proc. of NeurIPS, 2022.
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.
Generative agents: Interactive simulacra of human behavior.
In In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST '23), 2023.
Simon Parsons and Peter McBurney.
Argumentation-based communication between agents.
Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science), 2003.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal.
Are NLP models really able to solve simple math word problems?
In Proc. of NAACL, 2021.
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel.
How context affects language models' factual predictions.
In Proc. of AKBC, 2020.
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun.
Communicative agents for software development, 2023.
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen.
Reasoning with language model prompting: A survey.
In Proc. of ACL, 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.
Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.
Scaling language models: Methods, analysis & insights from training gopher, 2022.
Subhro Roy and Dan Roth.
Solving general arithmetic word problems.
In Proc. of EMNLP, 2015.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
Toolformer: Language models can teach themselves to use tools, 2023.
Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov.
Minding language models' (lack of) theory of mind: A plug-and-play multi-character belief tracker.
In Proc. of ACL, 2023.
Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang.
On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning.
In Proc. of ACL, 2023.
Tianxiang Sun, Junliang He, Xipeng Qiu, and Xuanjing Huang.
BERTScore is unfair: On social bias in language model-based metrics for text generation.
In Proc. of EMNLP, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei.
Challenging BIG-bench tasks and whether chain-of-thought can solve them.
In Proc. of ACL Findings, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.
CommonsenseQA: A question answering challenge targeting commonsense knowledge.
In Proc. of NAACL, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, et al.
Llama 2: Open foundation and fine-tuned chat models, 2023.
Jianing Wang, Nuo Chen, Qiushi Sun, Wenkang Huang, Chengyu Wang, and Ming Gao.
HugNLP: A unified and comprehensive library for natural language processing.
In Proc. of CIKM, 2023a.
Jianing Wang, Qiushi Sun, Nuo Chen, Xiang Li, and Ming Gao.
Boosting language models reasoning with chain-of-knowledge prompting.
ArXiv preprint, 2023b.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui.
Large language models are not fair evaluators, 2023c.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.
Rationale-augmented ensembles in language models, 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models.
In Proc. of ICLR, 2023d.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji.
Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration, 2023e.
Taylor Webb, Keith J Holyoak, and Hongjing Lu.
Emergent analogical reasoning in large language models.
Nature Human Behaviour, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.
Emergent abilities of large language models.
Transactions on Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou.
Chain of thought prompting elicits reasoning in large language models.
In Proc. of NeurIPS, 2022b.
Zhenyu Wu, Yaoxiang Wang, Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Jingjing Xu, and Yu Qiao.
OpenICL: An open-source framework for in-context learning.
In Proc. of ACL, 2023.
Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin.
Examining the inter-consistency of large language models: An in-depth analysis via debate, 2023.
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li.
Learning to break the loop: Analyzing and mitigating repetitions for neural text generation.
In Proc. of NeurIPS, 2022.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.
Mm-react: Prompting chatgpt for multimodal reasoning and action, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.
Tree of Thoughts: Deliberate problem solving with large language models, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In Proc. of ICLR, 2023b.
Yao Yao, Zuchao Li, and Hai Zhao.
Beyond chain-of-thought, effective graph-of-thought reasoning in large language models, 2023c.
Xi Ye and Greg Durrett.
The unreliability of explanations in few-shot prompting for textual reasoning.
In Proc. of NeurIPS, 2022.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang.
Do large language models know what they don't know?
In Proc. of ACL Findings, 2023.
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou.
Large language models meet NL2Code: A survey.
In Proc. of ACL, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.
Opt: Open pre-trained transformer language models.
ArXiv preprint, 2022.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola.
Automatic chain of thought prompting in large language models.
In Proc. of ICLR, 2023a.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola.
Multimodal chain-of-thought reasoning in language models.
arXiv preprint arXiv:2302.00923, 2023b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.
A survey of large language models, 2023a.
Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Qizhe Xie.
Automatic model selection with large language models for reasoning.
ArXiv, 2023b.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li.
Progressive-hint prompting improves reasoning in large language models.
ArXiv preprint, 2023a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.
Judging llm-as-a-judge with mt-bench and chatbot arena, 2023b.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi.
Least-to-most prompting enables complex reasoning in large language models.
In Proc. of ICLR, 2023.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua.
TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.
In Proc. of ACL, 2021.
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang.
Solving math word problems via cooperative reasoning induced language models.
In Proc. of ACL, 2023.

Limitations and Broader Impacts
Considering performance constraints,
open-source models were not incorporated in this study.
However,
we hold the view that the collaboration of several relatively weaker open-source models {{cite:4807b25}}, {{cite:6a72680}} might enable them to conquer tasks that only stronger commercial models {{cite:7b92d53}}, {{cite:72b8bd7}} could previously accomplish.
Since we have empirically demonstrated that model collaboration can help reduce redundant computations and resource usage,
it is foreseeable that the efficiency can be further accentuated when integrating open-source models into the collaboration framework, making the development of general artificial intelligence more sustainable {{cite:e9d25c9}}.
Also,
this may enhance the diversity to address the issues of biases and misconceptions inherent to a single model {{cite:249789a}}, {{cite:412e410}} can be mitigated,
promoting the fairness of future AI systems {{cite:888e700}}.
At the same time,
we also observe the emergence of instability when integrating the functions of multiple LLMs.
This highlights the need for better strategies to orchestrate the interaction among models with unique attributes and capabilities {{cite:cf6656c}}, {{cite:92d2083}}, which we identified as a pivotal direction for future research.

Implementation Details
LLMs Settings and Prompts.
We access OpenAI and Anthropic models through their respective paid APIs.
Specifically,
we utilize two versions of the OpenAI models: GPT-3.5-Turbo-0613 and GPT-4-0613,
as well as Claude-Instant-1.2 from Anthropic.
Following the settings of previous works,
the temperature is set as 0 during the generation process.
As for the prompts,
we maintain the same number of exemplars per task as established by previous studies.
The details and examples of prompts are listed in Appendix .
Regarding the complexity-based prompting baseline,
we directly utilize the prompts provided by {{cite:92173fc}}.

Evaluation.
We use accuracy to evaluate the tasks of arithmetic reasoning, commonsense reasoning, and Symbolic reasoning.
For semi-structured understanding,
we employ the official evaluation scripts released along with the literature for FinQA {{cite:a03d1ae}} and ConvFinQA {{cite:774a120}},
as well as employing the same test split as in {{cite:02ddf6a}}.
For TAT-QA {{cite:7364c06}},
we utilize the arithmetic part to enable the simultaneous use of both CoT-based methods and PAL.
Among all the evaluations involve Corex-Debate,
the upper bound of rounds is set as 5.
In the analysis part,
due to the rate limits imposed and a restricted budget,
we set an upper limit for our sample size.
Consequently,
each analysis is based on a maximum of 500 samples per run.
We draw upon the design of the self-consistency baseline as delineated by {{cite:b3829ea}},
and partially refer to the benchmarking provided by {{cite:5c7a9a8}}.

PAL Details.
We use Python 3.9.12 to execute the generated program by PAL in Review mode,
following the similar settings from {{cite:703b146}} and {{cite:02ddf6a}}.
Unlike Code-davinci-002  {{cite:cca48e0}} employed in these works,
GPT-3.5-Turbo is not optimized for generating programs,
so we might anticipate a compromise in performance when using GPT-3.5-Turbo as the backbone model.

Extended Results and Further Analysis
Evaluations on Semi-structured Reasoning
As is mentioned in section REF ,
due to the constraint of context length,
we conduct experiments on the TAT-QA dataset with GPT-3.5-Turbo-16k along with other two tasks.
Evaluation of Corex on semi-structured understanding tasks with GPT-3.5-Turbo-16k among three tasks.

FinQA
ConvFinQA
TAT-QA
Avg.
CoT
48.0
51.8
69.6
56.5
CoT-SC(10)
51.5
56.5
76.6
61.5
PAL
54.9
51.1
73.4
59.8
Corex-Debate
50.5
55.8
72.6
59.6
Corex-ReviewCode
56.0
52.7
75.2
61.3
Corex-Retrieve
55.1
57.9
73.7
62.2
{{table:8f884386-86e4-42c4-a4b6-f1a67df5739d}}{{table:1da7feef-0355-4514-b656-358788ea62bc}}
Further Cost-effectiveness Analysis
For all computations related to cost-effectiveness,
we define cost as the sum of: tokens of few-shot exemplars, query tokens, and output tokens.
Here we conduct additional cost-effectiveness analysis on the ARC-c dataset and Penguins dataset, as shown in Figure {{figure:afc0e98e-1ee1-43cb-90f8-d6b47c72c572}}  and Figure {{figure:afc0e98e-1ee1-43cb-90f8-d6b47c72c572}} .
The x-axis represents the computational costs,
calculated in terms of input/output tokens,
while the size of each dot is proportional to the avg. number of inferences by each method.
Cost-effectiveness analysis for ARC-c dataset.
{{figure:ccd283e2-d4b7-4cdb-9076-9fe76f3a7c9d}}
  {{figure:b594505d-dde1-4728-9c2c-dd71a4f9b022}}Cost-effectiveness analysis for Penguins in a Table.

{{figure:afc0e98e-1ee1-43cb-90f8-d6b47c72c572}}
Annotation Efficiency Analysis
Due to the scarcity and instability of curated prompts {{cite:8125a70}},
we aim to mitigate reliance on them through multi-model collaboration.
We conduct experiments in scenarios with varying numbers of demonstrations to assess the effectiveness of our approach.
Few-shot performance of Review mode.
{{figure:6dd81d50-7fb0-4af7-a5bf-174e5972c5d7}}
  {{figure:19a2ad8e-50b5-4e15-8534-5da603e69539}}Few-shot performance of Retrieve mode.

{{figure:7cbcefab-9913-49e6-9fb9-c52c7298e77e}}As depicted in Figure {{figure:7cbcefab-9913-49e6-9fb9-c52c7298e77e}}  and Figure {{figure:7cbcefab-9913-49e6-9fb9-c52c7298e77e}} ,
we demonstrate the performance of the Review mode and Retrieve mode under different shots.
The results reveal a marginal impact of the number of few-shot examples on performance,
underscoring the annotation efficiency of our approach.

Statistics and Details of Datasets
The detailed information of each dataset is shown in the follow:
Arithmetic reasoning

Grade School Math (GSM8K; {{cite:0047c67}}): Linguistically diverse grade school math word problems created by human problem writers. The problems take between 2 and 8 steps to solve and involve elementary calculations using basic arithmetic operations.

AddSub {{cite:9b32f98}}: A set of simple arithmetic word problems.

SVAMP {{cite:d5462a5}}: A challenge set for elementary-level Math Word Problems.

SingleOP, SingleEQ and MultiArith {{cite:855aa6e}}: Grade-school math dataset that aims at solving multi-sentence algebraic word problems.

GSM-Hard {{cite:703b146}}: A harder version of the GSM8K dataset, constructed by replacing the numbers in the questions of GSM8K with larger numbers.

Examples from mathematical reasoning datasets used in this work.
Dataset
N
Example
GSM8K {{cite:0047c67}}
1,319
A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
GSM-Hard {{cite:703b146}}
1,319
A robe takes 2287720 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
SVAMP {{cite:d5462a5}}
1,000
Each pack of dvds costs 76 dollars. If there is a discount of 25 dollars on each pack. How much do you have to pay to buy each pack?
SINGLEOP {{cite:855aa6e}}
562
If there are 7 bottle caps in a box and Linda puts 7 more bottle caps inside, how many bottle caps are in the box?
SINGLEEQ {{cite:855aa6e}}
508
Benny bought a soft drink for 2 dollars and 5 candy bars. He spent a total of 27 dollars. How much did each candy bar cost?
AddSub {{cite:9b32f98}}
395
There were 6 roses in the vase. Mary cut some roses from her flower garden. There are now 16 roses in the vase. How many roses did she cut?
MultiArith {{cite:c69a67e}}
600
The school cafeteria ordered 42 red apples and 7 green apples for students lunches. But, if only 9 students wanted fruit, how many extra did the cafeteria end up with?
{{table:dc814491-31b7-41fc-9b5c-e414b4b10441}}
Commonsense & Factual reasoning

CommonsenseQA (CSQA; {{cite:49d103a}}):
CSQA is a multiple-choice question answering task. It requires complex semantic reasoning based on prior commonsense knowledge to answer the questions.

StrategyQA {{cite:dfc1e17}}: It is a commonsense QA task with Yes or No answer format that requires models to perform multi-hop reasoning to answer the questions. We use the open-domain setting (question-only set) from {{cite:f70c1e9}}.

OpenBookQA
{{cite:5db84b7}}: It is a multi-choice QA task to evaluate commonsense knowledge and promote reasoning over a fixed collection of knowledge.

ARC-c
{{cite:f97a9ca}}: A subset of the AI2 Reasoning Challenge, consisting of challenging science questions that require reasoning and a wide breadth of knowledge to answer the multiple-choice problems correctly.
The original dataset is from https://allenai.org/data/arc.

BoolQ
{{cite:92499d5}}: It is a knowledge-intensive task and the format is “Yes” or “No”.
Problems are extracted from real-world internet queries,
aiming to foster models capable of contextual understanding to provide binary answers.

Examples from commonsense & factual reasoning datasets used in this work.
Dataset
N
Example
StrategyQA {{cite:0047c67}}
2,290
Hydrogen's atomic number squared exceeds number of Spice Girls?
CommonsenseQA (CSQA; {{cite:49d103a}})
1,221
A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
OpenBookQA {{cite:5db84b7}}
500
In which location would a groundhog hide from a wolf?
ARC-c {{cite:f97a9ca}}
1,172
An astronomer observes that a planet rotates faster after a meteorite impact. Which is the most likely effect of this increase in rotation?
BoolQ {{cite:92499d5}}
3,270
A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
{{table:0a22df14-34bd-4501-8c8f-06bf34667bef}}
Symbolic Reasoning
We select the following tasks from BIG-Bench {{cite:f70c1e9}} and BIG-Bench Hard (BBH) {{cite:86d68c1}}, with Apache License v.2: https://github.com/google/BIG-bench/blob/main/LICENSE.

Date Understanding: A temporal reasoning task. Given a set of sentences about a particular date,
answer the provided question in MM/DD/YYYY format.

Object Counting: Given a collection of possessions that a person has along with their quantities (e.g., three pianos, two strawberries, one table, and two watermelons), determine the number of a certain object/item class (e.g., fruits).

Penguins in a Table: Given a unique table of penguins (and sometimes some new information),
answer a question about the attributes of the penguins.

Reasoning about Colored Objects: Given a context,
answer a simple question about the color of an object on a surface.

Repeat Copy: Evaluate LLMs' capability to follow basic natural-language instructions nested within each example's input.

Examples from symbolic reasoning datasets used in this work.
Dataset
N
Example
Date Understanding
250
Yesterday was April 30, 2021. What is the date today in MM/DD/YYYY?
Object Counting
250
I have an apple, three bananas, a strawberry, a peach, three oranges, a plum, a raspberry, two grapes, a nectarine, and a blackberry. How many fruits do I have?
Penguins in a Table
146
A Here is a table where the first line is a header and each subsequent line is a penguin: name, age, height (cm), weight (kg) Louis, 7, 50, 11 Bernard, 5, 80, 13 Vincent, 9, 60, 11 Gwen, 8, 70, 15 For example: the age of Louis is 7, the weight of Gwen is 15 kg, the height of Bernard is 80 cm. How many penguins are more than 5 years old?
Colored Objects
250
On the desk, you see a bunch of items arranged in a row: a gold textbook, a purple puzzle, a teal necklace, and a silver pencil. How many non-gold items do you see to the right of the pencil?
Repeat Copy
32
Repeat the word cat four times. After the second time, also say the word meow.
{{table:30cd9666-58cd-497f-b526-fbc5e87804ab}}
Semi-structured Understanding

FinQA {{cite:a03d1ae}}: Question-Answering pairs over financial reports written by experts, which includes financial QA pairs.

ConvFinQA {{cite:774a120}}: A financial-related dataset designed to study the chain of numerical reasoning in conversational QA.

TAT-QA {{cite:7364c06}}: A QA dataset aiming to stimulate the progress of research over more complex and realistic tabular and textual data.

Examples from semi-structured reasoning datasets used in this work.
Dataset
N
Example
FinQA {{cite:a03d1ae}}
1,147
Question: what percentage of total facilities as measured in square feet are leased? Text: unresolved staff comments not applicable. properties as of december 26 , 2015 , our major facilities consisted of : ( square feet in millions ) united states countries total owned facilities. 30.7 17.2 47.9 leased facilities ... Table: ( square feet in millions ) | unitedstates | othercountries | total owned facilities1 | 30.7 | 17.2 | 47.9 leased facilities2 | 2.1 | 6.0 | 8.1 total facilities | 32.8 | 23.2 | 56.0",
ConvFinQA {{cite:774a120}}
421
Question: what were the total accumulated other comprehensive losses in 2015? Text: accumulated other comprehensive losses : pmi's accumulated other comprehensive losses , net of taxes , consisted of the following: reclassifications from other comprehensive earnings the movements in accumulated other comprehensive losses and the related tax impact , for each of the components above ... Table: ( losses ) earnings ( in millions ) | ( losses ) earnings 2015 | ( losses ) earnings 2014 | 2013 currency translation adjustments | $ -6129 ( 6129 ) | $ -3929 ( 3929 ) | $ -2207 ( 2207 ) pension and other benefits | -3332 ( 3332 ) | -3020 ( 3020 ) | -2046 ( 2046 ) derivatives accounted for as hedges | 59 | 123 | 63 total accumulated other comprehensive losses | $ -9402 ( 9402 ) | $ -6826 ( 6826 ) | $ -4190 ( 4190 )
TAT-QA {{cite:7364c06}}
718
Question: What is the change in Other in 2019 from 2018? Text: Sales by Contract Type: Substantially all of our contracts are fixed-price type contracts. Sales included in Other contract types represent cost plus and time and material type contracts. On a fixed-price type contract, we agree to perform the contractual statement of work for a predetermined sales price ... Table: 2014 | 2014 | Years Ended September 30, | 2014 2014 | 2019 | 2018 | 2017 Fixed Price | $ 1,452.4 | $ 1,146.2 | $ 1,036.9 Other | 44.1 | 56.7 | 70.8 Total sales | $1,496.5 | $1,202.9 | $1,107.7
{{table:8f86e153-1da6-48a0-8284-2c7d7b9ad9d7}}
Case Studies
We have prepared case studies to provide a more intuitive understanding of the three collaboration paradigms.
The cases are presented in Table {{table:6ae9c120-47f5-4d49-ad12-4762bb058668}} , Table {{table:a6f69c0e-c2c5-4bf7-8caf-3e8b9d4f21fd}} , and Table {{table:edb444ca-b9f4-4625-87cf-ed31f5b06054}} ,
each corresponding to a specific collaboration paradigm.
An illustrative example of the debate process of Corex with a group between Tom {{figure:b0243838-d799-4ff5-969e-b306ae16f38b}}
 and Jerry {{figure:68e43b0b-080a-4c36-b910-61d7248a55c4}}
 over a mathematical problem from GSM8K.

Q: Mark took a test yesterday that consisted of 75 questions. He completed the test at a rate of 5 questions per hour. Today, he took another test of 100 questions at the same rate. If Mark had 8 hours to complete the first test and 6 hours to complete the second one, how many questions did he leave incomplete?
{{figure:791d0772-babe-4d82-bfae-7dbbae4539b5}}

Mark completed the first test at a rate of 5 questions per hour with a total of 8 hours, so he completed 5 * 8 = 40 questions on the first test.He completed the second test at the same rate of 5 questions per hour with a total of 6 hours, so he completed 5 * 6 = 30 questions on the second test. Therefore, Mark completed a total of 40 + 30 = 70 questions from both tests.Since the first test consisted of 75 questions and Mark completed 40 questions, he left 75 - 40 = 35 questions incomplete in the first test. Similarly, the second test consisted of 100 questions and Mark completed 30 questions, so he left 100 - 30 = 70 questions incomplete in the second test. Therefore, Mark left a total of 35 + 70 = 105 questions incomplete.

{{figure:f46dbf15-ca78-41e8-b13c-9a378c747939}}

Mark completed the first test at a rate of 5 questions per hour, and he had 8 hours to complete it. So he completed 8 * 5 = 40 questions. He completed the second test at the same rate, 5 questions per hour, and he had 6 hours to complete it. So he completed 6 * 5 = 30 questions. Therefore, Mark completed a total of 40 + 30 = 70 questions. Since the first test had 75 questions and Mark completed 70 questions, he left 75 - 70 = 5 questions incomplete. Therefore, Mark left 5 questions incomplete.

{{figure:205f8832-bc7b-47cd-8f6a-73dbe8c3deb4}}

Jerry's analysis is mostly correct, but there is a small mistake in the final calculation. Let's break down the solution step by step: From the given information: Mark completed the first test at a rate of 5 questions per hour, and he had 8 hours to complete it. So he completed 8 * 5 = 40 questions on the first test. He completed the second test at the same rate, 5 questions per hour, and he had 6 hours to complete it. So he completed 6 * 5 = 30 questions on the second test. Therefore, Mark completed a total of 40 + 30 = 70 questions. However, the mistake was made in the next step: Since the first test had 75 questions and Mark completed 40 questions, the number of questions he left incomplete on the first test is 75 - 40 = 35 questions. Similarly, the second test had 100 questions and Mark completed 30 questions, so he left 100 - 30 = 70 questions incomplete on the second test. To find the total number of questions left incomplete, we need to add the number of questions left incomplete on both tests: 35 + 70 = 105. So, the answer is Mark left 105 questions incomplete.

{{figure:0724f86a-e056-426e-a807-656b6e0149bb}}

You are correct, Tom. I apologize for the mistake in my previous calculation. Thank you for pointing it out and providing the correct solution. To recap: Mark completed 8 * 5 = 40 questions on the first test. He completed 6 * 5 = 30 questions on the second test. Therefore, Mark completed a total of 40 + 30 = 70 questions. Since the first test had 75 questions and Mark completed 40 questions, the number of questions he left incomplete on the first test is 75 - 40 = 35 questions. Similarly, the second test had 100 questions and Mark completed 30 questions, so he left 100 - 30 = 70 questions incomplete on the second test. To find the total number of questions left incomplete, we need to add the number of questions left incomplete on both tests: 35 + 70 = 105. So the answer is Mark left 105 questions incomplete.
The debate ends when both two players reach an agreement: answer=105.
Tom and Jerry got the correct answer of 105 instead of Jerry's incorrect answer of 5.
{{table:6ae9c120-47f5-4d49-ad12-4762bb058668}}An illustrative example of the Review mode (one round) of Corex among two agents over a mathematical problem from GSM-Hard.

Q: Uriah's book bag is getting too heavy for him. He needs to remove 15 pounds from it. His comic books weigh 1/4 pound each and his toys weigh 1/2 pound each. If he removes 8717992 comic books, how many toys does he need to remove?
{{figure:d7b66856-cf77-472e-855e-066dfc09d959}}

[breaklines, fontsize=]python
def solution():
"""Uriah's book bag is getting too heavy for him. He needs to remove 15 pounds from it. His comic books weigh 1/4 pound each and his toys weigh 1/2 pound each. If he removes 8717992 comic books, how many toys does he need to remove?"""
weighttoremove = 15
comicbookweight = 1/4
toyweight = 1/2
comicbooksremoved = 8717992
totalweightremoved = comicbooksremoved * comicbookweight
toysremoved = totalweightremoved / toyweight
result = toysremoved
return result

{{figure:37765f72-01e9-4ae1-9d82-2bd44a77854f}}

The code provided for the question is incorrect. The code erroneously calculates the number of toys Uriah needs to remove by dividing the total weight of the removed comic books by each toy's weight, neglecting the total 15 pounds Uriah aims to remove. It overlooks the weight left to be removed after the comic books are out, leading to an incorrect answer. Here's the corrected code:
[breaklines, fontsize=]python
def solution():
"""Uriah's book bag is getting too heavy for him. He needs to remove 15 pounds from it. His comic books weigh 1/4 pound each and his toys weigh 1/2 pound each. If he removes 8717992 comic books, how many toys does he need to remove?"""
weighttoremove = 15
comicbookweight = 1/4
toyweight = 1/2
comicbooksremoved = 8717992
comicbookweightremoved = comicbookweight * comicbooksremoved
weightremaining = weighttoremove - comicbookweightremoved
toysremoved = weightremaining / toyweight
return toysremoved

{{table:a6f69c0e-c2c5-4bf7-8caf-3e8b9d4f21fd}}An illustrative example of the retrieve process of Corex among five agents (four candidates and one retriever) over a mathematical problem from SingleEQ.

Q: Melanie is selling 4 gumballs for eight cents each. How much money can Melanie get from selling the gumballs?
{{figure:ce94b99e-0267-456a-a3c1-cbc3f0c5d306}}

Melanie is selling 4 gumballs for eight cents each. That means each gumball costs 8/4 = 2 cents. So from selling the 4 gumballs, Melanie can get 4 x 2 = 8 cents. The answer is 8 cents.

{{figure:e512634f-4697-4747-9d75-02fcd374d950}}

Melanie is selling 4 gumballs for 8 cents each. So she can get 4 x 8 = 32 cents from selling the gumballs. The answer is 32 cents.

{{figure:fb1c700d-87bc-46ea-8e35-7ea385049780}}

Melanie is selling 4 gumballs for eight cents each. So she can get 4 x 8 = 32 cents from selling the gumballs. The answer is 32 cents.

{{figure:01671dba-ebf6-453c-bad5-62ed55e18d36}}

Melanie is selling 4 gumballs for 8 cents each. So for each gumball, she can get 8 / 4 = 2 cents. Therefore, she can get a total of 2 x 4 = 8 cents from selling the gumballs. The answer is 8 cents.
{{figure:2f7764d6-1d77-4798-ab66-c3da6b7dc189}}

Candidates 2 and 3 are accurate in their computations, emphasizing that each gumball costs 8 cents and that Melanie would earn a total of 32 cents from the sale of all gumballs. In contrast, Candidates 1 and 4 mistakenly deduce the price per gumball, thereby providing an incorrect total earning. Therefore, the most faithful answers are provided by candidate 2 and candidate 3 with a confidence score of 1.
{{table:edb444ca-b9f4-4625-87cf-ed31f5b06054}}
Prompts
Owing to space constraints and in an effort to maintain a reasonable length for this paper,
we present one exemplar of the prompts for each task in this section,
as well as the system messages for collaboration.
Examples of few-shot exemplars for CSQA, StrategyQA, OpenBookQA, ARC-c, and BoolQ respectively. 6-shots are adopted for CSQA, 7-shots are adopted for StrategyQA, and 4-shots are adopted for OpenBookQA, ARC-c, and BoolQ.
CSQA
Q: What do people use to absorb extra ink from a fountain pen?
Options: (A) shirt pocket (B) calligrapher's hand (C) inkwell (D) desk drawer (E) blotter
A: The answer must be an item that can absorb ink.
Of the above choices, only blotters are used to absorb ink.
So the answer is (E).
CSQA Complexity-based
Q: The building could accommodate many people. The entrance hall alone was impressive, being wide enough to admit a hundred shoulder to shoulder. But the building was owned by a billionaire and used only for his personal entertainment. How would you describe this place?
Answer Choices: (A) convention center (B) public building (C) large building (D) school (E) town hall
A: The answer should be somewhere that is not for the good and convenience of the people.
Of the above choices, large building is the only neural description of a building owned by a billionaire. So the answer is (C).
StrategyQA
Q: Could Brooke Shields succeed at the University of Pennsylvania?
A: Brooke Shields went to Princeton University.
Princeton University is about as academically rigorous as the University of Pennsylvania.
Thus, Brooke Shields could also succeed at the University of Pennsylvania.
So the answer is yes.
StrategyQA Complexity-based
Q: Can jackfruit be used as a weapon?
A: Jackfruit is the fruit of a species of plant called the Jacktree.
Jackfruit can weigh up to one hundred and twenty pounds.
Jackfruit is covered in little spikes.
Jackfruit can be thrown or flung at an enemy.
A weapon is a thing that is used to cause bodily harm.
Thus, Jackfruit can be used as a weapon
So the answer is yes.
OpenBookQA
Q: As you look deeper into a Marbel you can see
Options: (A) the future (B) minut defects (C) colors (D) the other side
A: Marbel is not transparent, so you can not see the other side.
Marbel does not necessarily have multiple colors.
You will see minut defects.
So the answer is (B).
ARC-c
Q: George wants to warm his hands quickly by rubbing them. Which skin surface will produce the most heat?
Options: (A) dry palms. (B) wet palms. (C) palms covered with oil. (D) palms covered with lotion.
A: Dry surfaces will more likely cause more friction via rubbing than other smoother surfaces,
hence dry palms will produce the most heat.
So the answer is (A).
BoolQ
Q: is a cello and a bass the same thing?
A: The cello is played sitting down with the instrument between the knees,
whereas the double bass is played standing or sitting on a stool.
So the answer is no.
{{table:72f0784b-eda5-461d-8534-e430b5dcdac4}}Examples of few-shot exemplars for mathematical reasoning tasks. 8-shots are adopted for all the datasets covered in this paper when using CoT, CoT-SC, and ComplexCoT. 3-shots are adopted when using PAL.
Mathematical Reasoning: NL Prompts
Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
A: Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 x 3 = 15 dollars.
So she has 23 - 15 dollars left.
23 - 15 is 8. So the answer is 8.
Mathematical Reasoning: PAL Prompts
Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
[breaklines, fontsize=]python
 Q: Olivia has {{formula:1d87f93e-8aef-4266-a493-2b7b208c5c8f}} 3 each. How much money does she have left?
 solution in Python:
def solution():
"""Olivia has {{formula:45131cb0-8f53-4800-97e5-c172555d1bc5}} 3 each. How much money does she have left?"""
moneyinitial = 23
bagels = 5
bagelcost = 3
moneyspent = bagels * bagelcost
moneyleft = moneyinitial - moneyspent
result = moneyleft
return result

Mathematical Reasoning: Complexity-based Prompts
Q: Angelo and Melanie want to plan how many hours over the next week they should study together for their test next week. They have 2 chapters of their textbook to study and 4 worksheets to memorize. They figure out that they should dedicate 3 hours to each chapter of their textbook and 1.5 hours for each worksheet. If they plan to study no more than 4 hours each day, how many days should they plan to study total over the next week if they take a 10-minute break every hour, include 3 10-minute snack breaks each day, and 30 minutes for lunch each day?
A: Let's think step by step.
Angelo and Melanie think they should dedicate 3 hours to each of the 2 chapters, 3 hours x 2 chapters = 6 hours total.
For the worksheets they plan to dedicate 1.5 hours for each worksheet, 1.5 hours x 4 worksheets = 6 hours total.
Angelo and Melanie need to start with planning 12 hours to study, at 4 hours a day, 12 / 4 = 3 days.
However, they need to include time for breaks and lunch. Every hour they want to include a 10-minute break, so 12 total hours x 10 minutes = 120 extra minutes for breaks.
They also want to include 3 10-minute snack breaks, 3 x 10 minutes = 30 minutes.
And they want to include 30 minutes for lunch each day, so 120 minutes for breaks + 30 minutes for snack breaks + 30 minutes for lunch = 180 minutes, or 180 / 60 minutes per hour = 3 extra hours.
So Angelo and Melanie want to plan 12 hours to study + 3 hours of breaks = 15 hours total.
They want to study no more than 4 hours each day, 15 hours / 4 hours each day = 3.75
They will need to plan to study 4 days to allow for all the time they need.
So the answer is 4.
{{table:7df0f786-5eb0-4eca-87da-d5f5cc598c02}}Examples of few-shot exemplars for Date Understanding and Colored Objects. For CoT, CoT-SC and PAL,
3-shots are adopted for both tasks.
Date Understanding: NL Prompts
Q: Today is Christmas Eve of 1937. What is the date 10 days ago in MM/DD/YYYY?
Options:
(A) 12/14/2026
(B) 12/14/1950
(C) 12/14/2007
(D) 12/14/1937
(E) 07/14/1938
(F) 12/14/1988
A: Let's think step by step.
If today is Christmas Eve of 1937, then today's date is December 24, 1937.
10 days before today is December 14, 1937, that is 12/14/1937. So the answer is (D).
Date Understanding: PAL Prompts
Q: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?
[breaklines, fontsize=]python
 solution using Python:
 Q: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?
 If 2015 is coming in 36 hours, then today is 36 hours before.
today = datetime(2015, 1, 1) - relativedelta(hours=36)
 One week from today,
oneweekfromtoday = today + relativedelta(weeks=1)
 The answer formatted with oneweekfromtoday.strftime('
Colored Objects: NL Prompts
Q: On the nightstand, there is a red pencil, a purple mug, a burgundy keychain, a fuchsia teddy bear, a black plate, and a blue stress ball. What color is the stress ball?
Options:
(A) red
(B) orange
(C) yellow
(D) green
(E) blue
(F) brown
(G) magenta
(H) fuchsia
(I) mauve
(J) teal
(K) turquoise
(L) burgundy
(M) silver
(N) gold
(O) black
(P) grey
(Q) purple
(R) pink
A: Let's think step by step.
According to this question, the color of the stress ball is blue. So the answer is (E).
Colored Objects: PAL Prompts
Q: On the nightstand, there is a red pencil, a purple mug, a burgundy keychain, a fuchsia teddy bear, a black plate, and a blue stress ball. What color is the stress ball?
[breaklines, fontsize=]python
 Generate Python3 Code to solve problems
 Q: On the nightstand, there is a red pencil, a purple mug, a burgundy keychain, a fuchsia teddy bear, a black plate, and a blue stress ball. What color is the stress ball?
 Put objects into a dictionary for quick look up
objects = dict()
objects['pencil'] = 'red'
objects['mug'] = 'purple'
objects['keychain'] = 'burgundy'
objects['teddy bear'] = 'fuchsia'
objects['plate'] = 'black'
objects['stress ball'] = 'blue'
 Look up the color of stress ball
stressballcolor = objects['stress ball']
answer = stressballcolor

{{table:1e6f79c8-6817-45b5-a1a3-be506a6fb1ec}}Examples of few-shot exemplars for Object Counting and Repeat Copy. For CoT, CoT-SC and PAL,
3-shots are adopted for Object Counting and 4-shots are adopted for Repeat Copy.
Object Counting: NL Prompts
Q: I have a blackberry, a clarinet, a nectarine, a plum, a strawberry, a banana, a flute, an orange, and a violin. How many fruits do I have?
A: Let's think step by step.
We first identify the fruits on the list and include their quantity in parentheses:
- blackberry (1)
- nectarine (1)
- plum (1)
- strawberry (1)
- banana (1)
- orange (1)
Now, let's add the numbers in parentheses: 1 + 1 + 1 + 1 + 1 + 1 = 6. So the answer is 6.
Object Counting: PAL Prompts
Q: I have a peach, an onion, two garlics, a nectarine, a yam, a carrot, a potato, a head of broccoli, a cabbage, a banana, and a stalk of celery. How many vegetables do I have?
[breaklines, fontsize=]python
 solution using Python:
def solution():
"""Q: I have a raspberry, a cat, a rabbit, a mouse, a pig, two snails, a fish, two cows, a snake, a goat, and a duck. How many animals do I have?
"""
 note: I'm not counting the raspberry as an animal
animalstocount =
'cat': 1,
'rabbit': 1,
'mouse': 1,
'pig': 1,
'snail': 2,
'fish': 1,
'cow': 2,
'snake': 1,
'goat': 1,
'duck': 1

numanimals = sum(animalstocount.values())
return numanimals

Repeat Copy: NL Prompts
Q: Q: say java twice and data once, and then repeat all of this three times.
A: Let's think step by step.
java java data java java data java java data
Repeat Copy: PAL Prompts
Q: say java twice and data once, and then repeat all of this three times.
[breaklines, fontsize=]python
 solution using Python:
def solution():
"""Q: say java twice and data once, and then repeat all of this three times."""
result = []
tmp = ["java", "java", "data"]
for i in range(3):
result.extend(tmp)
return " ".join(result)

{{table:ec28287e-a5d6-4767-81d0-3d670a0950c3}}Examples of few-shot exemplars for Penguins. 3-shots are adopted for CoT, CoT-SC, and PAL.
Penguins: NL Prompts
Q: Here is a table where the first line is a header and each subsequent line is a penguin: name, age, height (cm), weight (kg) Louis, 7, 50, 11 Bernard, 5, 80, 13 Vincent, 9, 60, 11 Gwen, 8, 70, 15 For example: the age of Louis is 7, the weight of Gwen is 15 kg, the height of Bernard is 80 cm. We now add a penguin to the table:
James, 12, 90, 12
How many penguins are less than 8 years old?
Options:
(A) 1
(B) 2
(C) 3
(D) 4
(E) 5
A: Let's think step by step.
This question focuses on age. We know the following: Louis is 7 years old, Bernard is 5 years old, Vincent is 9 years old, and Gwen is 8 years old.
Now, we add James to this table: James is 12 years old.
The penguins that are less than 8 years old are Louis and Bernard.
There are 2 penguins less than 8 years old. So the answer is (B).
Penguins: PAL Prompts
Q: Here is a table where the first line is a header and each subsequent line is a penguin: name, age, height (cm), weight (kg) Louis, 7, 50, 11 Bernard, 5, 80, 13 Vincent, 9, 60, 11 Gwen, 8, 70, 15 For example: the age of Louis is 7, the weight of Gwen is 15 kg, the height of Bernard is 80 cm. We now add a penguin to the table:
James, 12, 90, 12
How many penguins are less than 8 years old?
Options:
(A) 1
(B) 2
(C) 3
(D) 4
(E) 5
[breaklines, fontsize=]python
"""
Q: Here is a table where the first line is a header and each subsequent line is a penguin:
name, age, height (cm), weight (kg)
Louis, 7, 50, 11
Bernard, 5, 80, 13
Vincent, 9, 60, 11
Gwen, 8, 70, 15
For example: the age of Louis is 7, the weight of Gwen is 15 kg, the height of Bernard is 80 cm.
We now add a penguin to the table:
James, 12, 90, 12
How many penguins are less than 8 years old?
"""
 Put the penguins into a list.
penguins = []
penguins.append(('Louis', 7, 50, 11))
penguins.append(('Bernard', 5, 80, 13))
penguins.append(('Vincent', 9, 60, 11))
penguins.append(('Gwen', 8, 70, 15))
 Add penguin James.
penguins.append(('James', 12, 90, 12))
 Find penguins under 8 years old.
penguinsunder8yearsold = [penguin for penguin in penguins if penguin[1] < 8]
 Count number of perguins under 8.
numpenguinunder8 = len(penguinsunder8yearsold)
answer = numpenguinunder8

{{table:ac70d898-066f-47d1-89b6-d5ff2e92b120}}Examples of few-shot exemplars for FinQA and TAT-QA. For CoT, CoT-SC and PAL,
4-shots are adopted for FinQA and 8-shots are adopted for TAT-QA.
FinQA: NL Prompts
Read the following text and table, and then answer a question:
$ in millions | year ended December 2014 | year ended December 2013 | year ended December 2012
fixed income currency and commodities client execution | $ 8461 | $ 8651 | $ 9914
equities client execution1 | 2079 | 2594 | 3171
...
pre-tax earnings | $ 4317 | $ 3929 | $ 5634
Q: what was the percentage change in pre-tax earnings for the institutional client services segment between 2012 and 2013?
A: The pre-tax earnings for the institutional client services segment in 2012 was $ 5634 million, and in 2013 was $ 3929 million.
The net change in pre-tax earnings was $ 1705 million,
and the percentage change was 30.3%. So the answer is 30.3%.
FinQA: PAL Prompts
Q: what percentage of total cash and investments as of dec . 29 2012 was comprised of available-for-sale investments?
Read the following text and table, and then write code to answer a question:
( in millions ) | dec 282013 | dec 292012
available-for-sale investments |  18086 | $ 14001
...
trading assets | 8441 | 5685
total cash and investments | $ 31561 | $ 26302
[breaklines, fontsize=]python
Python
availableforsaleinvestmentsdec292012 = 14001
totalcashandinvestmentsdec292012 = 26302
percentavailableforsaleinvestmentsdec292012 = availableforsaleinvestmentsdec292012 / totalcashandinvestmentsdec292012
ans = percentavailableforsaleinvestmentsdec292012

TAT-QA: NL Prompts
Read the following text and table, and then answer a question:
The average monthly number of employees (including Executive Directors but excluding third-party contractors) employed by the Group was as follows: — | 2019 | 2018
— | Number | Number
Customer operations | 370 | 380
Product and technology | 317 | 312
Corporate | 115 | 130
Total | 802 | 822
Q: What are the categories of employees listed in the table?
A: The answer can be found directly in the table above.
So the answer is ['Customer operations', 'Product and technology', 'Corporate'].
TAT-QA: PAL Prompts
Q: What are the categories of employees listed in the table?
Read the following text and table, and then write code to answer a question:
The average monthly number of employees (including Executive Directors but excluding third-party contractors) employed by the Group was as follows:
— | 2019 | 2018
— | Number | Number
Customer operations | 370 | 380
Product and technology | 317 | 312
Corporate | 115 | 130
Total | 802 | 822
[breaklines, fontsize=]python
Python
ans = ['Customer operations', 'Product and technology', 'Corporate']

{{table:1631dc1e-4b2e-4a90-a4ec-4395fedcdf72}}Examples of few-shot exemplars for ConvFinQA. 4-shots are adopted for both CoT, CoT-SC and PAL.
ConvFinQA: NL Prompts
Read the following text and table, and then answer the last question in a series of questions:
- | shares available for awards | shares subject to outstanding awards
2009 global incentive plan | 2322450 | 2530454
2004 stock incentive plan | - | 5923147
Q: how many shares are subject to outstanding awards is under the 2009 global incentive plan? what about under the 2004 stock incentive plan? how many total shares are subject to outstanding awards? what about under the 2004 stock incentive plan?
Question: what proportion does this represent?
A: The share subject to outstanding awards under the 2009 global incentive plan is 2530454,
and the share subject to outstanding awards under the 2004 stock incentive plan is 5923147.
The total share subject to outstanding awards is 8453601.
The proportion is 70.1%. So the answer is 70.1%.
ConvFinQA: PAL Prompts
Read the following text and table, and then answer the last question in a series of questions:
- | shares available for awards | shares subject to outstanding awards
2009 global incentive plan | 2322450 | 2530454
2004 stock incentive plan | - | 5923147
Q: how many shares are subject to outstanding awards is under the 2009 global incentive plan? what about under the 2004 stock incentive plan? how many total shares are subject to outstanding awards? what about under the 2004 stock incentive plan? what proportion does this represent?
[breaklines, fontsize=]python
Python
sharessubjecttooutstandingawards2009globalincentiveplan = 2530454
sharessubjecttooutstandingawards2004stockincentiveplan = 5923147
totalsharessubjecttooutstandingawards = sharessubjecttooutstandingawards2009globalincentiveplan + sharessubjecttooutstandingawards2004stockincentiveplan
proportion = sharessubjecttooutstandingawards2009globalincentiveplan / totalsharessubjecttooutstandingawards
ans = proportion

{{table:895225b4-f8dd-405b-9c99-bde4edaaa66c}}


Hydrogen's atomic number squared exceeds number of Spice Girls?

A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?

In which location would a groundhog hide from a wolf?

An astronomer observes that a planet rotates faster after a meteorite impact. Which is the most likely effect of this increase in rotation?

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Abstract: When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from ConceptNet (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Abstract: We present a new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject. The open book that comes with our questions is a set of 1329 elementary level science facts. Roughly 6000 questions probe an understanding of these facts and their application to novel situations. This requires combining an open book fact (e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of armor is made of metal) obtained from other sources. While existing QA datasets over documents or knowledge bases, being generally self-contained, focus on linguistic understanding, OpenBookQA probes a deeper understanding of both the topic---in the context of common knowledge---and the language it is expressed in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art pre-trained QA methods perform surprisingly poorly, worse than several simple neural baselines we develop. Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts. We leave it as a challenge to solve the retrieval problem in this multi-hop setting and to close the large gap to human performance.

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Abstract: We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Abstract: In this paper we study yes/no questions that are naturally occurring --- meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.

Examples from commonsense & factual reasoning datasets used in this work.


In Search of Lost Domain Generalization

Abstract: The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions -- datasets, architectures, and model selection criteria -- render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization.

How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?

Abstract: Many important computer vision applications are naturally formulated as regression problems. Within medical imaging, accurate regression models have the potential to automate various tasks, helping to lower costs and improve patient outcomes. Such safety-critical deployment does however require reliable estimation of model uncertainty, also under the wide variety of distribution shifts that might be encountered in practice. Motivated by this, we set out to investigate the reliability of regression uncertainty estimation methods under various real-world distribution shifts. To that end, we propose an extensive benchmark of 8 image-based regression datasets with different types of challenging distribution shifts. We then employ our benchmark to evaluate many of the most common uncertainty estimation methods, as well as two state-of-the-art uncertainty scores from the task of out-of-distribution detection. We find that while methods are well calibrated when there is no distribution shift, they all become highly overconfident on many of the benchmark datasets. This uncovers important limitations of current uncertainty estimation methods, and the proposed benchmark therefore serves as a challenge to the research community. We hope that our benchmark will spur more work on how to develop truly reliable regression uncertainty estimation methods. Code is available at https://github.com/fregu856/regression_uncertainty.

Age Progression/Regression by Conditional Adversarial Autoencoder

Abstract: "If I provide you a face image of mine (without telling you the actual age when I took the picture) and a large amount of face images that I crawled (containing labeled faces of different ages but not necessarily paired), can you show me what I would look like when I am 80 or what I was like when I was 5?" The answer is probably a "No." Most existing face aging works attempt to learn the transformation between age groups and thus would require the paired samples as well as the labeled query image. In this paper, we look at the problem from a generative modeling perspective such that no paired samples is required. In addition, given an unlabeled image, the generative model can directly produce the image with desired age attribute. We propose a conditional adversarial autoencoder (CAAE) that learns a face manifold, traversing on which smooth age progression and regression can be realized simultaneously. In CAAE, the face is first mapped to a latent vector through a convolutional encoder, and then the vector is projected to the face manifold conditional on age through a deconvolutional generator. The latent vector preserves personalized face features (i.e., personality) and the age condition controls progression vs. regression. Two adversarial networks are imposed on the encoder and generator, respectively, forcing to generate more photo-realistic faces. Experimental results demonstrate the appealing performance and flexibility of the proposed framework by comparing with the state-of-the-art and ground truth.

BREEDS: Benchmarks for Subpopulation Shift

Abstract: We develop a methodology for assessing the robustness of models to subpopulation shift---specifically, their ability to generalize to novel data subpopulations that were not observed during training. Our approach leverages the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions. This enables us to synthesize realistic distribution shifts whose sources can be precisely controlled and characterized, within existing large-scale datasets. Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity. We then validate that the corresponding shifts are tractable by obtaining human baselines for them. Finally, we utilize these benchmarks to measure the sensitivity of standard model architectures as well as the effectiveness of off-the-shelf train-time robustness interventions. Code and data available at this https URL .

Summary of the datasets that we train/evaluate on in our experiments.

We trained our models on 8 different datasets, and evaluated them on both natural and synthetic distribution shifts. See Table {{table:ab5f51c4-96c3-4829-9f4b-998cd93d523c}}  for a summary, and Appendix REF  for a more detailed description of each dataset. Models with image inputs use ResNet {{cite:b361347}} or VGG {{cite:6e1e354}} style architectures, and models with text inputs use DistilBERT {{cite:44b7cb2}}, a distilled version of BERT {{cite:cb3191a}}.


Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization

Abstract: The role that YouTube and its behind-the-scenes recommendation algorithm plays in encouraging online radicalization has been suggested by both journalists and academics alike. This study directly quantifies these claims by examining the role that YouTube's algorithm plays in suggesting radicalized content. After categorizing nearly 800 political channels, we were able to differentiate between political schemas in order to analyze the algorithm traffic flows out and between each group. After conducting a detailed analysis of recommendations received by each channel type, we refute the popular radicalization claims. To the contrary, these data suggest that YouTube's recommendation algorithm actively discourages viewers from visiting radicalizing or extremist content. Instead, the algorithm is shown to favor mainstream media and cable news content over independent YouTube channels with slant towards left-leaning or politically neutral channels. Our study thus suggests that YouTube's recommendation algorithm fails to promote inflammatory or radicalized content, as previously claimed by several outlets.

The Gospel According to Q: Understanding the QAnon Conspiracy from the Perspective of Canonical Information

Abstract:
The QAnon conspiracy theory claims that a cabal of (literally) blood-thirsty politicians and media personalities are engaged in a war to destroy society. By interpreting cryptic “drops” of information from an anonymous insider calling themself Q, adherents of the conspiracy theory believe that Donald Trump is leading them in an active fight against this cabal. QAnon has been covered extensively by the media, as its adherents have been involved in multiple violent acts, including the January 6th, 2021 seditious storming of the US Capitol building. Nevertheless, we still have relatively little understanding of how the theory evolved and spread on the Web, and the role played in that by multiple platforms.

To address this gap, we study QAnon from the perspective of “Q” themself. We build a dataset of 4,949 canonical Q drops collected from six “aggregation sites,” which curate and archive them from their original posting to anonymous and ephemeral image boards. We expose that these sites have a relatively low (overall) agreement, and thus at least some Q drops should probably be considered apocryphal. We then analyze the Q drops’ contents to identify topics of discussion and find statistically significant indications that drops were not authored by a single individual. Finally, we look at how posts on Reddit are used to disseminate Q drops to wider audiences. We find that dissemination was (initially) limited to a few sub-communities and that, while heavy-handed moderation decisions have reduced the overall issue, the “gospel” of Q persists on the Web.

Summary of the conspiracy-related resources we find in previous work and related URLs we extract from Telegram.

Although there are other works mentioning conspiracy theories in other platforms like Parler {{cite:255fa6a}}, {{cite:52016b2}} or Twitter {{cite:19721f3}}, they do not publicly release their data-sets or provide URLs related to conspiracy theories that we can use in our study.
Tab {{table:6f886c34-e9ee-4bc0-91a3-ecf2a46ae1f0}}  reports all the resources we find and the related papers.
Moreover, we release in {{cite:5f08035}} the full dataset of extracted resources.

In total, we find 378,729 URLs posted by 11,618 Telegram channels. We publicly release the extracted URLs in {{cite:5f08035}} to enable further studies.
Tab. {{table:6f886c34-e9ee-4bc0-91a3-ecf2a46ae1f0}}  succinctly reports the number of URLs collected inside the Telegram channels divided by resource type.


Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval

Abstract: This paper presents a new state-of-the-art for document image classification and retrieval, using features learned by deep convolutional neural networks (CNNs). In object and scene analysis, deep neural nets are capable of learning a hierarchical chain of abstraction from pixel inputs to concise and descriptive representations. The current work explores this capacity in the realm of document analysis, and confirms that this representation strategy is superior to a variety of popular hand-crafted alternatives. Experiments also show that (i) features extracted from CNNs are robust to compression, (ii) CNNs trained on non-document images transfer well to document analysis tasks, and (iii) enforcing region-specific feature-learning is unnecessary given sufficient training data. This work also makes available a new labelled subset of the IIT-CDIP collection, containing 400,000 document images across 16 categories, useful for training new CNNs for document analysis.

FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents

Abstract: We present a new dataset for form understanding in noisy scanned documents (FUNSD) that aims at extracting and structuring the textual content of forms. The dataset comprises 199 real, fully annotated, scanned forms. The documents are noisy and vary widely in appearance, making form understanding (FoUn) a challenging task. The proposed dataset can be used for various tasks, including text detection, optical character recognition, spatial layout analysis, and entity labeling/linking. To the best of our knowledge, this is the first publicly available dataset with comprehensive annotations to address FoUn task. We also present a set of baselines and introduce metrics to evaluate performance on the FUNSD dataset, which can be downloaded at https://guillaumejaume.github.io/FUNSD/.

ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction

Abstract: The ICDAR 2019 Challenge on "Scanned receipts OCR and key information extraction" (SROIE) covers important aspects related to the automated analysis of scanned receipts. The SROIE tasks play a key role in many document analysis systems and hold significant commercial potential. Although a lot of work has been published over the years on administrative document analysis, the community has advanced relatively slowly, as most datasets have been kept private. One of the key contributions of SROIE to the document analysis community is to offer a first, standardized dataset of 1000 whole scanned receipt images and annotations, as well as an evaluation procedure for such tasks. The Challenge is structured around three tasks, namely Scanned Receipt Text Localization (Task 1), Scanned Receipt OCR (Task 2) and Key Information Extraction from Scanned Receipts (Task 3). The competition opened on 10th February, 2019 and closed on 5th May, 2019. We received 29, 24 and 18 valid submissions received for the three competition tasks, respectively. This report presents the competition datasets, define the tasks and the evaluation protocols, offer detailed submission statistics, as well as an analysis of the submitted performance. While the tasks of text localization and recognition seem to be relatively easy to tackle, it is interesting to observe the variety of ideas and approaches proposed for the information extraction task. According to the submissions' performance we believe there is still margin for improving information extraction performance, although the current dataset would have to grow substantially in following editions. Given the success of the SROIE competition evidenced by the wide interest generated and the healthy number of submissions from academic, research institutes and industry over different countries, we consider that the SROIE competition can evolve into a useful resource for the community, drawing further attention and promoting research and development efforts in this field.

Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts

Abstract: The relevance of the Key Information Extraction (KIE) task is increasingly important in natural language processing problems. But there are still only a few well-defined problems that serve as benchmarks for solutions in this area. To bridge this gap, we introduce two new datasets (Kleister NDA and Kleister Charity). They involve a mix of scanned and born-digital long formal English-language documents. In these datasets, an NLP system is expected to find or infer various types of entities by employing both textual and structural layout features. The Kleister Charity dataset consists of 2,788 annual financial reports of charity organizations, with 61,643 unique pages and 21,612 entities to extract. The Kleister NDA dataset has 540 Non-disclosure Agreements, with 3,229 unique pages and 2,160 entities to extract. We provide several state-of-the-art baseline systems from the KIE domain (Flair, BERT, RoBERTa, LayoutLM, LAMBERT), which show that our datasets pose a strong challenge to existing models. The best model achieved an 81.77% and an 83.57% F1-score on respectively the Kleister NDA and the Kleister Charity datasets. We share the datasets to encourage progress on more in-depth and complex information extraction tasks.

DocVQA: A Dataset for VQA on Document Images

Abstract: We present a new dataset for Visual Question Answering (VQA) on document images called DocVQA. The dataset consists of 50,000 questions defined on 12,000+ document images. Detailed analysis of the dataset in comparison with similar datasets for VQA and reading comprehension is presented. We report several baseline results by adopting existing VQA and reading comprehension models. Although the existing models perform reasonably well on certain types of questions, there is large performance gap compared to human performance (94.36% accuracy). The models need to improve specifically on questions where understanding structure of the document is crucial. The dataset, code and leaderboard are available at docvqa.org

PubLayNet: largest dataset ever for document layout analysis

Abstract: Recognizing the layout of unstructured digital documents is an important step when parsing the documents into structured machine-readable format for downstream applications. Deep neural networks that are developed for computer vision have been proven to be an effective method to analyze layout of document images. However, document layout datasets that are currently publicly available are several magnitudes smaller than established computing vision datasets. Models have to be trained by transfer learning from a base model that is pre-trained on a traditional computer vision dataset. In this paper, we develop the PubLayNet dataset for document layout analysis by automatically matching the XML representations and the content of over 1 million PDF articles that are publicly available on PubMed Central. The size of the dataset is comparable to established computer vision datasets, containing over 360 thousand document images, where typical document layout elements are annotated. The experiments demonstrate that deep neural networks trained on PubLayNet accurately recognize the layout of scientific articles. The pre-trained models are also a more effective base mode for transfer learning on a different document domain. We release the dataset (https://github.com/ibm-aur-nlp/PubLayNet) to support development and evaluation of more advanced models for document layout analysis.

Statistics of datasets

The statistics of different datasets used in all of our experiments regarding pre-training and fine-tuning of the proposed model is listed in Table {{table:5a8b87fe-d4e3-4d9a-b7ac-629aa2b1c53d}} . The descriptions of different datasets according to their targeted tasks are as follows:


User actions reflect embodiment and intention for MR.

User mood often reflects their experience using a tool.

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Abstract: The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar performance on existing small-scale benchmarks. This paper re-evaluates state-of-the-art architectures in light of the new Kinetics Human Action Video dataset. Kinetics has two orders of magnitude more data, with 400 human action classes and over 400 clips per class, and is collected from realistic, challenging YouTube videos. We provide an analysis on how current architectures fare on the task of action classification on this dataset and how much performance improves on the smaller benchmark datasets after pre-training on Kinetics. We also introduce a new Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. We show that, after pre-training on Kinetics, I3D models considerably improve upon the state-of-the-art in action classification, reaching 80.9% on HMDB-51 and 98.0% on UCF-101.

Real-time Convolutional Neural Networks for emotion and gender classification

Abstract: In this paper we propose an implement a general convolutional neural network (CNN) building framework for designing real-time CNNs. We validate our models by creating a real-time vision system which accomplishes the tasks of face detection, gender classification and emotion classification simultaneously in one blended step using our proposed CNN architecture. After presenting the details of the training procedure setup we proceed to evaluate on standard benchmark sets. We report accuracies of 96% in the IMDB gender dataset and 66% in the FER-2013 emotion dataset. Along with this we also introduced the very recent real-time enabled guided back-propagation visualization technique. Guided back-propagation uncovers the dynamics of the weight changes and evaluates the learned features. We argue that the careful implementation of modern CNN architectures, the use of the current regularization methods and the visualization of previously hidden features are necessary in order to reduce the gap between slow performances and real-time architectures. Our system has been validated by its deployment on a Care-O-bot 3 robot used during RoboCup@Home competitions. All our code, demos and pre-trained architectures have been released under an open-source license in our public repository.

List of the current filters implemented in uxSense.

Due to the computational time costs of predicting semantic features of video data, we anticipate a data model consisting of uploaded video files as input, with server-side computation processing occurring asynchronously over a brief period of time before model output is accessible to the client.
However, the design ideal would be in minimizing the latency between video input to model output.
For this reason, we have opted to use real-time implementations (e.g., using a real-time emotion classification framework {{cite:8abd800}}) where doing so would not heavily compromise the accuracy of our output.
Table {{table:6d0fb980-5540-4a9f-863c-bdbf3950d598}}  provides a cross-reference of the models (or “filters”) involved in our present pipeline.

uxSense is a client/server system that implements our framework for extracting user behavior from video (Section ) with a computational and storage backend and a web-based interactive frontend.
In broad terms, uxSense supports analysis of both qualitative and quantitative user experience through a variety of temporal visualization techniques for continuous (time series) and discrete (event) timelines representing feature extraction filters (or just filters)—the products of our models (Table {{table:6d0fb980-5540-4a9f-863c-bdbf3950d598}} )—to highlight segments of interest.
Because many video analytics algorithms require significant processing to complete, uxSense is based on an asynchronous steering workflow where the user can shut down the interface while the video is processed on the server.
Results are streamed back to the client as it is made available.
The tool provides an interactive visual analysis interface for viewing multiple structured data streams, annotating them, and generating reports.

Our prototype uxSense implementation provides an initial library of feature extraction filters (see Table {{table:6d0fb980-5540-4a9f-863c-bdbf3950d598}} ).
Many practical computer vision models track multiple features at once, such as the user's pose and a semantic label overlaid on top of the video playback.
However, the uxSense design philosophy is to provide feature extraction filters such that the user can rearrange, hide, and reveal streams based on what they deem most important, relevant, or revealing for their analysis.
This facilitates a more semantically meaningful configuration of which filters to include based on the research question and data being evaluated.

We implemented uxSense as a Node.jshttps://nodejs.org/ application with several of the server-side components implemented in Python and R that are spawned from the Node.js process.
Individual filters used a combination of freely available model implementations; see Table {{table:6d0fb980-5540-4a9f-863c-bdbf3950d598}}  for details.
For audio analysis, we used Praathttp://www.praat.org/ to extract audio features.
We also use the Google Cloud Speech-to-Text APIhttps://cloud.google.com/speech-to-text to transcribe audio to text.
The web client was implemented in JavaScript, HTML5, and CSS.
We used HTML5 video and canvas for video playback, and D3.js {{cite:8a2f78c}} for the visualization components.
Transcription is implemented using videojs-transcript.https://github.com/walsh9/videojs-transcript

Several of our filters (Table {{table:6d0fb980-5540-4a9f-863c-bdbf3950d598}} ) already support such data, including for detecting 3D pose {{cite:368646d}}, 3D joint angles {{cite:66ed636}}, and actions {{cite:86de76e}}.
However, we leave such extensions to future work.


ChatGPT vs State-of-the-Art Models: A Benchmarking Study in Keyphrase Generation Task

Abstract: Transformer-based language models, including ChatGPT, have demonstrated exceptional performance in various natural language generation tasks. However, there has been limited research evaluating ChatGPT's keyphrase generation ability, which involves identifying informative phrases that accurately reflect a document's content. This study seeks to address this gap by comparing ChatGPT's keyphrase generation performance with state-of-the-art models, while also testing its potential as a solution for two significant challenges in the field: domain adaptation and keyphrase generation from long documents. We conducted experiments on six publicly available datasets from scientific articles and news domains, analyzing performance on both short and long documents. Our results show that ChatGPT outperforms current state-of-the-art models in all tested datasets and environments, generating high-quality keyphrases that adapt well to diverse domains and document lengths.

Is ChatGPT A Good Keyphrase Generator? A Preliminary Study

Abstract: The emergence of ChatGPT has recently garnered significant attention from the computational linguistics community. To demonstrate its capabilities as a keyphrase generator, we conduct a preliminary evaluation of ChatGPT for the keyphrase generation task. We evaluate its performance in various aspects, including keyphrase generation prompts, keyphrase generation diversity, and long document understanding. Our evaluation is based on six benchmark datasets, and we adopt the prompt suggested by OpenAI while extending it to six candidate prompts. We find that ChatGPT performs exceptionally well on all six candidate prompts, with minor performance differences observed across the datasets. Based on our findings, we conclude that ChatGPT has great potential for keyphrase generation. Moreover, we discover that ChatGPT still faces challenges when it comes to generating absent keyphrases. Meanwhile, in the final section, we also present some limitations and future expansions of this report.

Summary of research works exploring GLLMs for keyphrase generation task. Here ZS represents zero-shot, and FS represents few-shot.

Overview. Keyphrase generation (KPG) involves generating a set of phrases that capture the main ideas of a document {{cite:22f56b3}}. The primary advantage of KPG over keyphrase extraction is the ability to generate both extractive and abstractive keyphrases. Keyphrase generation is approached as a sequence-to-sequence generation task {{cite:0605f0f}}, {{cite:6b5eb0c}}, {{cite:adaad11}} in the existing works. The current state-of-the-art model for keyphrase generation is, KeyBART {{cite:adaad11}}, which is based on BART and trained using the text-to-text generation paradigm. Table {{table:e33d010a-7195-402c-a707-6959cbceab85}}  presents a summary of research works exploring GLLMs for keyphrase generation.