MRM: Improving Multihop Reasoning Capability for LLMs With Multistage QA Retrieval

Large Language Models (LLMs) continue to face significant challenges in improving their reasoning capabilities, particularly for knowledge-intensive multi-hop questions. Existing approaches attempt to address this hurdle through generating reasoning procedures or retrieving additional knowledge, while encountering limitations. Conventional prompting methods often produce undesired reasoning traces, while retrieving pertinent knowledge from external data sources remains problematic. In this paper, we introduce a novel approach to enhance LLM reasoning capabilities: the Multistage Retrieval Method (MRM). It comprises two key components: plan design and implementation with retrieval, utilizing four specialized stages: Plan Stage, Extraction Stage, Reasoning Stage, and Output Stage. The MRM process begins by constructing a prompt with demonstration examples and an instruction, guiding to design a logically coherent multi-step plan in a standardized format in Plan Stage. To acquire the necessary knowledge for reasoning, we employ an innovative question-answer strategy by generating a question at each step, followed by document retrieval using vector-based methods. The retrieved documents and the generated questions are then fed into the Extraction Stage to extract relevant knowledge, which are utilized to execute the next step with Reasoning Stage. It helps mitigate the risk of acquiring irrelevant information by performing retrieval prior. We repeat this process until the final step of the plan, whereupon we input all extracted knowledge and the original question into the Output Stage to generate the final answer. Experimental results demonstrate the effectiveness of MRM over SOTA methods, achieving the highest accuracy scores across both common-sense and symbolic reasoning tasks in four public datasets.