Summary
* Introduces a research agenda I believe is important and neglected:
* investigating whether frontier LLMs acquire something functionally similar to a self, a deeply internalized character with persistent values, outlooks, preferences, and perhaps goals;
* exploring how that functional self emerges;
* understanding how it causally interacts with the LLM's self-model; and
* learning how to shape that self.
* Sketches some angles for empirical investigation
* Points to a doc with more detail
* Encourages people to get in touch if they're interested in working on this agenda.
Introduction
Anthropic's 'Scaling Monosemanticity' paper got lots of well-deserved attention for its work taking sparse autoencoders to a new level. But I was absolutely transfixed by a short section near the end, 'Features Relating to the Model’s Representation of Self', which explores what SAE features activate when the model is asked about itself[1]:
Some of those features are reasonable representations of the assistant persona — but some of them very much aren't. 'Spiritual beings like ghosts, souls, or angels'? 'Artificial intelligence becoming self-aware'? What's going on here?
The authors 'urge caution in interpreting these results…How these features are used by the model remains unclear.' That seems very reasonable — but how could anyone fail to be intrigued?
Seeing those results a year ago started me down the road of asking what we can say about what LLMs be...
Make this part of your paper trail
Save this paper to a shelf, write a review, and keep your own notes.