Epistemic status: These questions seem useful to me, but I'm biased. I'm interested in your thoughts on any portion you read.
If our first AGI is based on current LLMs and alignment strategies, is it likely to be adequately aligned? Opinions and intuitions vary widely.
As a lens to analyze this question, let's consider such a proto-AGI reasoning about its goals. This scenario raises questions that can be addressed empirically in current-gen models.
1. Scenario/overview:
SuperClaude is super nice
Anthropic has released a new Claude Agent, quickly nicknamed SuperClaude because it's impressively useful for longer tasks. SuperClaude thinks a lot in the course of solving complex problems with many moving parts. It's not brilliant, but it can crunch through work and problems, roughly like a smart and focused human. This includes a little better long-term memory, and reasoning to find and correct some of its mistakes. This is more or less the default path as seen from 2025. By at least a weak definition, AGI has arrived. We might call such a system a Long-Horizon agentic Learning Language Model, LHLLM.
SuperClaude is super "nice" in its starting state. It's helpful, harmless, and honest (HHH), the targets of its constitutional RLAIF alignment training. It reasons about ethics much like an intelligent and highly ethical human. We'll assume that SuperClaude has "nice" thoughts, and there's no mesaoptimizer hidden in its weights after initial training.
But now there’s a ri...
Make this part of your paper trail
Save this paper to a shelf, write a review, and keep your own notes.