Please consider add more logical reasoning papers in the arxiv paper and github repo #35
Description
Hi,
Thanks for sharing the good survey paper and github repo about logical reasoning.
Furthermore, we have more papers working on logical reasoning. Here are our new papers for logical reasoning data augmentation, prompt augmentation and evaluation. Please consider to add those papers into your arXiv paper and github repo if you find them related. Thanks a lot.
Logic-Driven Data Augmentation and Prompt Augmentation
We present an AMR-based logic-driven data augmentation for contrastive learning to improve discriminative language model's logical reasoning performance and we also use AMR-based data augmentation method to augment the prompt which help GPT-4 achieved #1 on the ReClor leaderboard (One of the hardest logical reasoning reading comprehension dataset, the data was collected from LSAT and GMAT) and we also achieved better performance than other baseline models on different logical reasoning reading comprehension tasks and natural language inference tasks. Here is the details for the paper.
[LLM@IJCAI 2023] Contrastive Learning with Logic-driven Data Augmentation for Logical Reasoning over Text [Paper link]
The full version has been accepted by [The findings of ACL 2024] "Abstract Meaning Representation-Based Logic-Driven Data Augmentation for Logical Reasoning" [Paper link] [Source code] [Model weights] [Leaderboard].
Logical Data Augmentation for Abductive reasoning
[The findings of ACL 2022] ""AbductionRules: Training Transformers to Explain Unexpected Inputs"" [Paper link] [Source code]. The proposed dataset AbductionRules have been collected by LogiTorch, ReasoningNLP, Prompt4ReasoningPapers and OpenAI/Evals.
Out-of-Distribution Logical Reasoning Evaluation and Prompt Augmentation for Enhancing OOD Logical Reasoning
We present a systematically out-of-distribution evaluation on logical reasoning tasks. We presented three new more robust logical reasoning datasets ReClor-Plus, LogiQA-Plus and LogiQAv2-Plus which are basically constructed from ReClor, LogiQA and LogiQAv2 from the changes of option's order and forms. We found simply using chain-of-thought prompting will not increase models' performance on the out-of-distribution scenario while using our AMR-based logic-driven data augmentation to augment prompt can increase large language models' performance on out-of-distribution logical reasoning tasks. The three datasets have been collected by OpenAI/Evals.
[LLM@IJCAI 2023] "A Systematic Evaluation of Large Language Models on Out-of-Distribution Logical Reasoning Tasks" [Paper link]
The full version named "Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning" has been accepted by ICONIP 2024. [Paper link] [Source code] [Dataset links].
Abstract Reasoning Evaluation Benchmark
[AGI@ICLR 2024] Large language models are not strong abstract reasoners [Paper link]
The full version has been accepted by [IJCAI 2024] Large Language Models Are Not Abstract Reasoners [Paper link] [Source code and evaluation platform]
A Empirical Study on Out-Of-Distribution Multi-Step Logical Reasoning
We find that pre-trained language models are not good at on robust multi-step logical reasoning tasks and one of the main reason is that there is limited amount of training sets for deeper multi-step logical reasoning. Therefore, we present a deeper large multi-step logical reasoning datasets named PARARULE-Plus. The dataset has also been collected by OpenAI/Evals.
[IJCLR-NeSy 2022] "Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation" [Paper link] [Source code] [Dataset links].
Explanation Generation by Multi-step Iterative LLM Post-training
Qiming Bao, Juho Leinonen, Alex Yuxuan Peng, Wanjun Zhong, Tim Pistotti, Alice Huang, Paul Denny, Michael Witbrock, Jiamou Liu. Exploring Iterative Enhancement for Improving Learnersourced Multiple-Choice Question Explanations with Large Language Models, AAAI/EAAI 2025 (AAAI 2025 Proceeding) [Paper link] [Source code]
Integrating Logic Programming with Large Language Model for Multi-step Reasoning
Zhongsheng Wang, Jiamou Liu, Qiming Bao, Hongfei Rong, Jingfeng Zhang. ChatLogic: Integrating Logic Programming with Large Language Models for Multi-step Reasoning, NucLeaR@AAAI 2024 [Paper link] [Source code]