This project was conducted as a part of a one-week research sprint for the Finnish Alignment Engineering Bootcamp. We would like to thank Vili Kohonen and Tyler Tracy for their feedback and guidance on the project.
TLDR
Research in AI control is tedious. To test the safety and usefulness of a new protocol, AI control researchers manually write the prompts to maximize the LLM’s performance on an evaluation. There are two problems with this process:
1. It’s time-consuming. Dead end protocols may be explored for too long, while the researcher manually changes prompts and reruns evaluations, only to finally determine the protocol is not effective.
2. Prompts are specialized for the model and context used during experimentation. An effective protocol with prompts which have been manually written for one dataset and one model may not transfer when the researcher tries it out with a different dataset or model.
DSPy optimization improves monitor safety across various audit budgets. The prompt-optimized monitors (green) achieve ~90% safety with a 1% audit budget, while baseline monitors fail to exceed 70% safety. Each box plot shows safety scores across 5 evaluation epochs on 169 APPS samples, with monitors trained on 200 samples from the ControlTax dataset. More detail.
We think prompt optimization can help with both problems, and we validate the process by using the DSPy library to optimize the prompt for a trusted monitor.
By giving natural language feedback to a reflection ...
Make this part of your paper trail
Save this paper to a shelf, write a review, and keep your own notes.