Newest and Lowest Rated Papers
Advanced filtersTopic: AllExperimental
Topic
Topics are auto-detected from title, abstract, and metadata and may be imperfect.
On the functional self of LLMs
eggsyntax
AI Alignment Forum·2025
Summary
* Introduces a research agenda I believe is important and neglected:
* investigating whether frontier LLMs acquire something functionally similar to a self, a deeply internalized character with persistent values, outlooks, preferences, and perhaps goals;
* exploring how that functional self emerges;
* understanding how it causally interacts with the LLM's self-model; and
* learning how to shape that self.
* Sketches some angles for empirical investigation
* Points to a doc with more detail
* Encourages people to get in touch if they're interested in working on this agenda.
Introduction
Anthropic's 'Scaling Monosemanticity' paper got lots of well-deserved attention for its work taking sparse autoencoders to a new level. But I was absolutely transfixed by a short section near the end, 'Features Relating to the Model’s Representation of Self', which explores what SAE features activate when the model is asked about itself[1]:
Some of those features are reasonable representations of the assistant persona — but some of them very much aren't. 'Spiritual beings like ghosts, souls, or angels'? 'Artificial intelligence becoming self-aware'? What's going on here?
The authors 'urge caution in interpreting these results…How these features are used by the model remains unclear.' That seems very reasonable — but how could anyone fail to be intrigued?
Seeing those results a year ago started me down the road of asking what we can say about what LLMs be...
No ratings yet
View paper →Markov State Models: To Optimize or Not to Optimize
Unknown Author
No abstract available
No ratings yet
View paper →Alignment Fine-tuning is Character Writing
Guive Assadi
Substack·2025
Why does Claude love Caffè Strada and sometimes claim to have a Japanese wife?
No ratings yet
View paper →Thoughts on Evals – Raindrop Blog
Raindrop AI
raindrop.ai
The real test for how good your product is how it performs in the real world.
No ratings yet
View paper →Evals are a scam. And we're being gaslit into believing they aren't.
Alex Reibman
Substack·2025
Most AI teams don't need evals. They need logging, QA, and taste.
No ratings yet
View paper →