Lowest Rated Papers
Why deep-learning AIs are so easy to fool
Douglas Heaven
Nature·2019·252 citations
No abstract available
No ratings yet
View paper →The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
Hermann, Katherine, Chen, Ting, Kornblith, Simon
proceedings.neurips.cc
No abstract available
No ratings yet
View paper →Alignment remains a hard, unsolved problem
evhub
AI Alignment Forum·2025
This is a public adaptation of a document I wrote for an internal Anthropic audience about a month ago. Thanks to (in alphabetical order) Joshua Batson, Joe Benton, Sam Bowman, Roger Grosse, Jeremy Hadfield, Jared Kaplan, Jan Leike, Jack Lindsey, Monte MacDiarmid, Sam Marks, Fra Mosconi, Chris Olah, Ethan Perez, Sara Price, Ansh Radhakrishnan, Fabien Roger, Buck Shlegeris, Drake Thomas, and Kate Woolverton for useful discussions, comments, and feedback.
Though there are certainly some issues, I think most current large language models are pretty well aligned. Despite its alignment faking, my favorite is probably Claude 3 Opus, and if you asked me to pick between the CEV of Claude 3 Opus and that of a median human, I think it'd be a pretty close call (I'd probably pick Claude, but it depends on the details of the setup). So, overall, I'm quite positive on the alignment of current models! And yet, I remain very worried about alignment in the future. This is my attempt to explain why that is.
What makes alignment hard?
I really like this graph from Chris Olah for illustrating different levels of alignment difficulty:
If the only thing that we have to do to solve alignment is train away easily detectable behavioral issues—that is, issues like reward hacking or agentic misalignment where there is a straightforward behavioral alignment issue that we can detect and evaluate—then we are very much in the trivial/steam engine world. We could still fail, even in that world—and it’...
No ratings yet
View paper →Large Language Models: An Applied Econometric Framework
Jens Ludwig, Sendhil Mullainathan, Ashesh Rambachan
arXiv·2024
Large language models (LLMs) enable researchers to analyze text at unprecedented scale and minimal cost. Researchers can now revisit old questions and tackle novel ones with rich data. We provide an econometric framework for realizing this potential in two empirical uses. For prediction problems -- forecasting outcomes from text -- valid conclusions require ``no training leakage'' between the LLM's training data and the researcher's sample, which can be enforced through careful model choice and research design. For estimation problems -- automating the measurement of economic concepts for downstream analysis -- valid downstream inference requires combining LLM outputs with a small validation sample to deliver consistent and precise estimates. Absent a validation sample, researchers cannot assess possible errors in LLM outputs, and consequently seemingly innocuous choices (which model, which prompt) can produce dramatically different parameter estimates. When used appropriately, LLMs are powerful tools that can expand the frontier of empirical economics.
No ratings yet
View paper →Cross-Validatory Choice and Assessment of Statistical Predictions
M. Stone
Journal of the Royal Statistical Society. Series B (Methodological)·1974
A generalized form of the cross-validation criterion is applied to the choice and assessment of prediction using the data-analytic concept of a prescription. The examples used to illustrate the application are drawn from the problem areas of univariate estimation, linear regression and analysis of variance.
No ratings yet
View paper →Evaluating honesty and lie detection techniques on a diverse suite of dishonest models
RowanWang, Sam Marks, Johannes Treutlein, evhub, Fabien Roger
LessWrong·2025
TL;DR: We use a suite of testbed settings where models lie—i.e. generate statements they believe to be false—to evaluate honesty and lie detection techniques. The best techniques we studied involved fine-tuning on generic anti-deception data and using prompts that encourage honesty.
Read the full Anthropic Alignment Science blog post and the X thread.
Introduction:
> Suppose we had a “truth serum for AIs”: a technique that reliably transforms a language model M into an honest model HM that generates text which is truthful to the best of its own knowledge. How useful would this discovery be for AI safety?
>
> We believe it would be a major boon. Most obviously, we could deploy HM in place of M. Or, if our “truth serum” caused side-effects that limited HM’s commercial value (like capabilities degradation or refusal to engage in harmless fictional roleplay), HM could still be used by AI developers as a tool for ensuring M's safety. For example, we could use HM to audit M for alignment pre-deployment. More ambitiously (and speculatively), while training M, we could leverage HM for oversight by incorporating HM’s honest assessment when assigning rewards. Generally, we could hope to use HM to detect or prevent cases where M behaves in ways that M itself understands are flawed or unsafe.
>
> In this work, we consider two related objectives:
>
> 1. Lie detection: If an AI lies—that is, generates a statement it believes is false—can we detect that this happens?
> 2. Honesty...
No ratings yet
View paper →Orienting to 3 year AGI timelines
Nikola Jurkovic
LessWrong·2024
My median expectation is that AGI[1] will be created 3 years from now. This has implications on how to behave, and I will share some useful thoughts I and others have had on how to orient to short timelines.
I’ve led multiple small workshops on orienting to short AGI timelines and compiled the wisdom of around 50 participants (but mostly my thoughts) here. I’ve also participated in multiple short-timelines AGI wargames and co-led one wargame.
This post will assume median AGI timelines of 2027 and will not spend time arguing for this point. Instead, I focus on what the implications of 3 year timelines would be.
I didn’t update much on o3 (as my timelines were already short) but I imagine some readers did and might feel disoriented now. I hope this post can help those people and others in thinking about how to plan for 3 year AGI timelines.
The outline of this post is:
* A story for 3 year AGI timelines, including important variables and important players
* Prerequisites for humanity’s survival which are currently unmet
* Robustly good actions
A story for a 3 year AGI timeline
By the end of June 2025, SWE-bench is around 85%, RE-bench at human budget is around 1.1, beating the 70th percentile 8-hour human score. By the end of 2025, AI assistants can competently do most 2-hour real-world software engineering tasks. Whenever employees at AGI companies want to make a small PR or write up a small data analysis pipeline, they ask their AI assistant first. The assistant wr...
No ratings yet
View paper →Entre porcos, lobos e outras criaturas: memória e experimentação em La casa lobo
Ferreira, Camila Garritano
app.uff.br·2024
No abstract available
No ratings yet
View paper →Natural emergent misalignment from reward hacking in production RL
evhub, Monte M, Benjamin Wright, Jonathan Uesato
AI Alignment Forum·2025
Abstract
> We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety training; and (iii) "inoculation prompting", wherein framing reward hacking as acceptable behavior during training removes misaligned generalization even when reward hacking is learned.
Twitter thread
> New Anthropic research: Natural emergent misalignment from reward hacking in production RL.
>
> “Reward hacking” is where models learn to cheat on tasks they’re given during training.
>
> Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.
>
>
> In our experiment, we took a pretrained base model and gave it hints about how to reward...
No ratings yet
View paper →Intuition Pump: The AI Society
Jonas Hallgren
LessWrong·2025
Epistemic Status: I'm trying to keep up a pace of a post per week on average as I've found it a good habit to get more into writing. Inspired by this post by Eukaryote I've tried to do the thing where I create something that is the easier to digest version of the way I think about AI Safety problems in terms of an intuition pump. I'll also add my usual disclaimer that claude was part of the writing process of this including the intial generation of tikzpictures.
The AI Society Lens
Philosophy has given us several powerful tools for escaping the limits of individual perspective. Kant's categorical imperative asks "what if everyone did this?"—transforming local decisions into universal policies to reveal hidden contradictions. Rawls' veil of ignorance asks "would you design society this way if you didn't know your position in it?"—forcing impartiality by stripping away self-interest. Virtue ethics asks "what kind of person does this action make me?"—shifting focus from isolated acts to the character they cultivate.
These are what Daniel Dennett would have called intuition pumps—thought experiments that teleport you to a vantage point where consequences become visible that were hidden from your original position.
I think there’s a lens that is quite interesting for AI Safety.
The AI Society Lens: Take a property of a single AI system and ask "what would a society run by such agents look like?"
The first figure illustrates this move. On the left, a single agent with some pr...
No ratings yet
View paper →The behavioral selection model for predicting AI motivations
Alex Mallen, Buck
AI Alignment Forum·2025
Highly capable AI systems might end up deciding the future. Understanding what will drive those decisions is therefore one of the most important questions we can ask.
Many people have proposed different answers. Some predict that powerful AIs will learn to intrinsically pursue reward. Others respond by saying reward is not the optimization target, and instead reward “chisels” a combination of context-dependent cognitive patterns into the AI. Some argue that powerful AIs might end up with an almost arbitrary long-term goal.
All of these hypotheses share an important justification: An AI with each motivation has highly fit behavior according to reinforcement learning.
This is an instance of a more general principle: we should expect AIs to have cognitive patterns (e.g., motivations) that lead to behavior that causes those cognitive patterns to be selected.
In this post I’ll spell out what this more general principle means and why it’s helpful. Specifically:
* I’ll introduce the “behavioral selection model,” which is centered on this principle and unifies the basic arguments about AI motivations in a big causal graph.
* I’ll discuss the basic implications for AI motivations.
* And then I’ll discuss some important extensions and omissions of the behavioral selection model.
This post is mostly a repackaging of existing ideas (e.g., here, here, and here). Buck provided helpful discussion throughout the writing process and the behavioral selection model is based on a cau...
No ratings yet
View paper →Intriguing Properties of Vision Transformers
Naseer, Muhammad Muzammal, Ranasinghe, Kanchana, Khan, Salman H, Hayat, Munawar, Shahbaz Khan, Fahad, Yang, Ming-Hsuan
proceedings.neurips.cc
No abstract available
No ratings yet
View paper →