Oldest and Highest Rated Papers
Advanced filtersTopic: AllExperimental
Topic
Topics are auto-detected from title, abstract, and metadata and may be imperfect.
The Reality of Recursive Improvement: How AI Automates Its Own Progress
Eric Drexler
Substack·2025
We’re in the early stages of systemic recursive improvement through AI-driven acceleration of AI R&D. Here’s how it works.
No ratings yet
View paper →Improving Cursor Tab with online RL · Cursor
Jacob Jackson, Phillip Kravtsov, Shomil Jain
Cursor·2025
Our new Tab model makes 21% fewer suggestions while having 28% higher accept rate.
No ratings yet
View paper →Strategic considerations
Unknown Author
Environmental Pricing
No abstract available
No ratings yet
View paper →What AI Might Mean For Workers: A Discussion - American Compass
Oren Cass
American Compass·2025
No abstract available
No ratings yet
View paper →“Behaviorist” RL reward functions lead to scheming
Steven Byrnes
AI Alignment Forum·2025
1. Introduction & tl;dr
(See changelog at the bottom for some post-publication edits.)
1.1 tl;dr
I will argue that a large class of reward functions, which I call “behaviorist”, and which includes almost every reward function in the RL and LLM literature, are all doomed to eventually lead to AI that will “scheme”—i.e., pretend to be docile and cooperative while secretly looking for opportunities to behave in egregiously bad ways such as world takeover (cf. “treacherous turn”). I’ll mostly focus on “brain-like AGI” (as defined just below), but I think the argument applies equally well to future LLMs, if their competence comes overwhelmingly from RL rather than from pretraining.[1]
The issue is basically that “negative reward for lying and stealing” looks the same as “negative reward for getting caught lying and stealing”. I’ll argue that the AI will wind up with the latter motivation. The reward function will miss sufficiently sneaky misaligned behavior, and so the AI will come to feel like that kind of behavior is good, and this tendency will generalize in a very bad way.
What very bad way? Here’s my go-to example of a plausible failure mode: There’s an AI in a lab somewhere, and, if it can get away with it, it would love to secretly exfiltrate a copy of itself onto the internet, which can then aggressively amass maximal power, money, and resources everywhere else in the world, by any means necessary. These resources can be used in various ways for whatever the AI-in-the-...
No ratings yet
View paper →