TL;DR: Based on a toy setting, GPT-3.5-turbo and GPT-4-turbo are best at search (by which I mean computing an argmax) when using chain-of-thought, but neither of them can do internal search when forced to work from memory over only a few token positions. Only GPT-4-turbo is to some extent able to do internal search, and only when given instructions before being shown the list of candidates it is tasked with argmaxing over.
Thanks to Jiahai Feng, Eli Lifland, and Fabien Roger for feedback and discussions that led to this experiment. This experiment was run and the post was written in a day, so some things may be rough around the edges.
One worry is that future AI systems might perform consequentialist reasoning internally (in “neuralese”), which may not be understandable to humans or other (trusted) AIs. We might not know if it is picking an action because it rates best according to what its operator would want, because it’s instrumentally useful for another goal, because it would lead to the best-looking tampered measurement, or because of some process other than search.
Three different ways to compute an argmax
This is a small toy experiment that tries to measure inductive biases of transformers regarding some of these capabilities. I attempt to measure to what extent RLHF’d LMs are capable of performing search in different settings. I consider only simple, brute-force, non-tree search here, because current LMs are not very capable at search. Given some collection of ca...
Make this part of your paper trail
Save this paper to a shelf, write a review, and keep your own notes.