Lowest Rated Papers
Dodging systematic human errors in scalable oversight
Geoffrey Irving
AI Alignment Forum·2025
Summary: Both our (UK AISI's) debate safety case sketch and Anthropic’s research agenda point at systematic human error as a weak point for debate. This post talks through how one might strengthen a debate protocol to partially mitigate this.
Not too many errors in unknown places
The complexity theory models of debate assume some expensive verifier machine M with access to a human oracle, such that
1. If we ran M in full, we’d get a safe answer
2. M is too expensive to run in full, meaning we need some interactive proof protocol (something like debate) to skip steps
Typically, M is some recursive tree computation, where for simplicity we can think of human oracle queries as occurring at the leaves of the tree. Key design elements encapsulated by the verifier machine M (rather than by the interactive proof protocol) are those related to the structure of the underlying debate, such as whether counterarguments are used, when the human oracle is consulted, and how the judges are trained.
Model the human oracle as type:
H:X→ΔY,
where X is the question type, Y is the answer type, and ΔY is distributions over Y (the humans are stochastic). Assumption (1) above then becomes
* Claim: M(H) is safe
This claim includes robustness to errors: M only has oracle access to H, so seeing through noise in H requires M to sample enough. It also includes robustness to systematic error: if H is systematically wrong, M(H) must sample different questions to dodge the wrongness.
This is ...
No ratings yet
View paper →The Work of Sexual Violence Reporting
Nelanthi Hewa
Journalism Practice·2025
No abstract available
No ratings yet
View paper →Distributional AGI Safety
Nenad Tomašev, Matija Franklin, Julian Jacobs, Sébastien Krier, Simon Osindero
arXiv·2025
AI safety and alignment research has predominantly been focused on methods for safeguarding individual AI systems, resting on the assumption of an eventual emergence of a monolithic Artificial General Intelligence (AGI). The alternative AGI emergence hypothesis, where general capability levels are first manifested through coordination in groups of sub-AGI individual agents with complementary skills and affordances, has received far less attention. Here we argue that this patchwork AGI hypothesis needs to be given serious consideration, and should inform the development of corresponding safeguards and mitigations. The rapid deployment of advanced AI agents with tool-use capabilities and the ability to communicate and coordinate makes this an urgent safety consideration. We therefore propose a framework for distributional AGI safety that moves beyond evaluating and aligning individual agents. This framework centers on the design and implementation of virtual agentic sandbox economies (impermeable or semi-permeable), where agent-to-agent transactions are governed by robust market mechanisms, coupled with appropriate auditability, reputation management, and oversight to mitigate collective risks.
No ratings yet
View paper →