Constitutional AI: Harmlessness from AI Feedback
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan
Abstract
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.
Make this part of your paper trail
Save this paper to a shelf, write a review, and keep your own notes.
Community Ratings
★3(1 rating)
3
★★★★★
1 rating
5
0
4
0
3
1
2
0
1
0
Community Reviews
@lukemperez
10/1/2026
NB: I might raise the score later. At the time of this writing (2026-10-01), it's the first paper in the queue to deep dive into AI training, safety, and governance policy. High-level overview for training AI models at scale while achieving metrics similar to, or exceeding, traditional training methods regarding harmlessness and helpfulness. As models increase in size and capability, human feedback becomes a bottleneck to scale. A set of principles, or constitution, can be used by a model specifically for training new models or supplementing human training. The approach makes intuitive sense. I wonder, however, how the principles can be defined in ways that are uniform or broadly enough accepted to be feasible without introducing bias in other directions. E.g., we don't want a new model to be racist, sexist, or toxic. As a principle no one would object. No doubt in the main these are relatively simple cases to identify during training. But at the margins what counts as any one of those will be shaped by time, place, culture, and individual taste. The data analysis relies mostly on creating Elo scores which is welcomed. But the description of how they created the analysis is thin and references mostly to other papers. Perhaps that's a citation form for the paper. But it left me asking a lot of questions about replication, mainly whether the analysis could be done with traditional research design (OLS / Probit / D-in-D /etc.). Not a dealbreaker but worth highlighting. On balance, recommended.