A new paper from Stanford researchers calls into question whether prompting large language models like GPT-3 and Codex to reason step-by-step truly unlocks logical reasoning abilities.
The paper tests a technique called “chain of thought” (CoT) prompting on tasks from BIG-Bench, a benchmark for evaluating AI capabilities. In CoT prompting, models are given examples walking through reasoning before answering a question. Surprisingly, the researchers found that prompting with illogical reasoning chains boosted performance nearly as much as logical reasoning chains.
“This demonstrates that completely illogical reasoning in the CoT prompts do not significantly harm the performance of the language model,” the authors write. “Our findings suggest that valid reasoning in prompting is not the chief driver of performance gains.”
Previously, the effectiveness of CoT prompting was touted as evidence that models can learn to logically reason. But this paper provides a reality check, showing that other factors enabled by CoT prompts likely contribute more to performance than logical reasoning itself.
The results raise important questions about what models are actually learning when prompted in certain ways. The authors suggest investigating what features prompts contain that aid model performance, whether increasing incorrectness in prompts impacts results, and when models produce inconsistent outputs.
If valid reasoning isn’t the key, what prompts model behaviors we interpret as reasoning? This research highlights the difficulty of determining if and how AI systems develop human-like logical reasoning. More work is needed to design prompts and benchmarks that accurately evaluate reasoning abilities.
Regardless, the study reinforces that prompting techniques like CoT can significantly boost model performance on complex tasks. This enables promising applications, even if the underlying mechanisms remain mysterious. Unlocking the full potential of large language models will require deepening our understanding of their abilities and limitations.