Lowest Rated Papers

Rating: Lowest Clear all

Advanced filtersTopic: All

Experimental

Topic

Topics are auto-detected from title, abstract, and metadata and may be imperfect.

Publication Date

Newest Oldest

Average Rating

Highest Lowest Clear

Forecasting With LLMs: Improved Generalization Through Feature Steering

Humzah Merchant, Bradford Levy

arXiv·2026

Successful forecasting involves identifying patterns between historical and future states of the world which generalize to future observations. We apply LLMs to a variety of forecasting tasks and inspect their internal states using sparse autoencoders to understand whether they appear to rely on time-specific pieces of knowledge versus generalizable patterns. Our analyses identify features associated with both time-aware reasoning and look-ahead-biased reasoning. We then apply the LLMs to an entirely different domain and intervene on these features. We find that amplifying time-awareness features substantially reduces look-ahead bias on forecasting prompts while preserving general reasoning performance. In contrast, steering the candidate look-ahead-bias features does not produce an effect. These results suggest that interpretable temporal features can be used to causally shift LLMs toward more historically grounded reasoning.

No ratings yet

View paper →

Reimagining Social Relations Through Care and Collective Joy: The Formation of Collectivity Among Volunteers Along the Lines of In/Formality

Michelle Brunßen, Sarah Jay, João Manuel de Oliveira

Journal of Community & Applied Social Psychology·2026·1 citations

ABSTRACT The neoliberal model of social organisation frames individualism as the dominant perspective to perceive the world and competition as its core practice. Interdependencies become marginalised, and ideas of social welfare and community are pushed aside. Mutual aid efforts not only aim to provide relief for resulting social inequalities but further seek to build new social relations. Adopting a feminist institutionalist approach, this paper aims to contribute to a move away from neoliberal structuring of social relations by analysing how collectivity among volunteers is formed. Based upon ethnographic fieldwork and eight interviews with volunteers in the context of Lesvos, Greece, this paper contextualises the role of collectivity among volunteers working in mutual aid organisations along the lines of in/formalities. It provides a deeper understanding of the role of affect by exploring practical tools through which affective collectivity is formed: care and collective joy. The paper argues that in/formalities are not distinct spheres but, much like the private‐public domains, constantly flow into and influence each other. While extending previous psychological research, the present paper shows that practices of care and collective joy, intertangled with in/formalities, offer possibilities to form affective mutual relationships between volunteers and are an integral part of collectivity formation.

No ratings yet

View paper →

Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

Matthew Vandergrift, Esraa Elelimy, Martha White

arXiv·2026

One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings. When running experiments, however, the goal of achieving high performance in the simulator can mutate into focusing exclusively on solving the simulator. To achieve high scores, researchers may adopt solutions exclusively meant for solving simulators, rather than learning while the agent is deployed outside a simulator. Solving simulators is also worthy of investigation, but it is a fundamentally different RL research question. In this paper, we argue that RL researchers need to distinguish between two use cases of simulators: solving simulators and using simulators as a proxy for learning in deployment. We first discuss how these two use-cases are importantly different, in terms of constraints on how the agent can use the simulator, which algorithms are appropriate, and which evaluation metrics are appropriate. We then highlight several issues and misleading conclusions that can occur by not making the distinction between these two settings clear, supported with examples and simple experiments. This work is a call to the community to begin clearly distinguishing how they are using simulators in their work, hopefully sparking further discussion on which empirical practices work best in each setting.

No ratings yet

View paper →

PreviousPage 515 of 613Next

Forager: a lightweight testbed for continual learning with partial observability in RL

Steven Tang, Xinze Xiong, Anna Hakhverdyan, Andrew Patterson, Jacob Adkins, Jiamin He, Esraa Elelimy, Parham Mohammad Panahi, Martha White, Adam White

arXiv·2026

In continual reinforcement learning (CRL), good performance requires never-ending learning, acting, and exploration in a big, partially observable world. Most CRL experiments have focused on loss of plasticity -- the inability to keep learning -- in one-off experiments where some unobservable non-stationarity is added to classic fully observable MDPs. Further, these experiments rarely consider the role of partial observability and the importance of CRL agents that use memory or recurrence. One potential reason for this focus on mitigating loss of plasticity without considering partial observability is that many partially-observable CRL environments are prohibitively expensive. In this paper, we introduce Forager, a light-weight partially-observable CRL environment with a constant memory footprint. We provide a set of experiments and sample tasks demonstrating that Forager is challenging for current CRL agents and yet also allows for in-depth study of those agents. We demonstrate that agents exhibit loss of plasticity, proposed mitigations can help, but that most useful is to leverage state construction. We conclude with a variant of Forager that generates an unending stream of new tasks to learn that clearly highlights the limitations of current CRL agents.

No ratings yet

View paper →

Deep Double Q-learning

Prabhat Nagarajan, Martha White, Marlos C. Machado

arXiv·2025

Double Q-learning is a classical control algorithm that mitigates the maximization bias of Q-learning. To do so, it explicitly trains two independent action-value functions and uses them to decouple action-selection and action-evaluation when computing bootstrap targets. Double DQN adapts target bootstrap decoupling to deep reinforcement learning (RL), but explicitly trains only a single action-value function and does not fully decouple its estimators. Consequently, the two estimators remain correlated, and overestimation persists. In this paper, we introduce Deep Double Q-learning (DDQL), a deep RL algorithm that explicitly trains two Q-functions through Double Q-learning. DDQL stabilizes training through a combination of techniques, including lower replay ratios, longer target network update intervals, and shared layers. Across 57 Atari 2600 games, DDQL improves aggregate performance over Double DQN, outperforming it on 47 games while further reducing overestimation. In addition, we study key design choices when adapting Double Q-learning to deep RL, including the network architecture, replay ratio, and minibatch sampling strategies.

No ratings yet

View paper →

Empirical Design in Reinforcement Learning

Andrew Patterson, Samuel Neumann, Martha White, Adam White

arXiv·2023

Empirical design in reinforcement learning is no small task. Running good experiments requires attention to detail and at times significant computational resources. While compute resources available per dollar have continued to grow rapidly, so have the scale of typical experiments in reinforcement learning. It is now common to benchmark agents with millions of parameters against dozens of tasks, each using the equivalent of 30 days of experience. The scale of these experiments often conflict with the need for proper statistical evidence, especially when comparing algorithms. Recent studies have highlighted how popular algorithms are sensitive to hyper-parameter settings and implementation details, and that common empirical practice leads to weak statistical evidence (Machado et al., 2018; Henderson et al., 2018). Here we take this one step further. This manuscript represents both a call to action, and a comprehensive resource for how to do good experiments in reinforcement learning. In particular, we cover: the statistical assumptions underlying common performance measures, how to properly characterize performance variation and stability, hypothesis testing, special considerations for comparing multiple agents, baseline and illustrative example construction, and how to deal with hyper-parameters and experimenter bias. Throughout we highlight common mistakes found in the literature and the statistical consequences of those in example experiments. The objective of this document is to provide answers on how we can use our unprecedented compute to do good science in reinforcement learning, as well as stay alert to potential pitfalls in our empirical design.

No ratings yet

View paper →

Efficient Off-Policy Learning for High-Dimensional Action Spaces

Fabian Otto, Philipp Becker, Ngo Anh Vien, Gerhard Neumann

arXiv·2024

Existing off-policy reinforcement learning algorithms often rely on an explicit state-action-value function representation, which can be problematic in high-dimensional action spaces due to the curse of dimensionality. This reliance results in data inefficiency as maintaining a state-action-value function in such spaces is challenging. We present an efficient approach that utilizes only a state-value function as the critic for off-policy deep reinforcement learning. This approach, which we refer to as Vlearn, effectively circumvents the limitations of existing methods by eliminating the necessity for an explicit state-action-value function. To this end, we leverage a weighted importance sampling loss for learning deep value functions from off-policy data. While this is common for linear methods, it has not been combined with deep value function networks. This transfer to deep methods is not straightforward and requires novel design choices such as robust policy updates, twin value function networks to avoid an optimization bias, and importance weight clipping. We also present a novel analysis of the variance of our estimate compared to commonly used importance sampling estimators such as V-trace. Our approach improves sample complexity as well as final performance and ensures consistent and robust performance across various benchmark tasks. Eliminating the state-action-value function in Vlearn facilitates a streamlined learning process, yielding high-return agents.

No ratings yet

View paper →

Position: Benchmarking is Limited in Reinforcement Learning Research

Scott M. Jordan, Adam White, Bruno Castro da Silva, Martha White, Philip S. Thomas

arXiv·2024

Novel reinforcement learning algorithms, or improvements on existing ones, are commonly justified by evaluating their performance on benchmark environments and are compared to an ever-changing set of standard algorithms. However, despite numerous calls for improvements, experimental practices continue to produce misleading or unsupported claims. One reason for the ongoing substandard practices is that conducting rigorous benchmarking experiments requires substantial computational time. This work investigates the sources of increased computation costs in rigorous experiment designs. We show that conducting rigorous performance benchmarks will likely have computational costs that are often prohibitive. As a result, we argue for using an additional experimentation paradigm to overcome the limitations of benchmarking.

No ratings yet

View paper →

Gradient Iterated Temporal-Difference Learning

Théo Vincent, Kevin Gerhardt, Yogesh Tripathi, Habib Maraqten, Adam White, Martha White, Jan Peters, Carlo D'Eramo

arXiv·2026

Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this paradigm implement a semi-gradient update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped estimate. While popular, this type of update is prone to divergence, as Baird's counterexample illustrates. Gradient TD methods were introduced to overcome this issue, but have not been widely used, potentially due to issues with learning speed compared to semi-gradient methods. Recently, iterated TD learning was developed to increase the learning speed of TD methods. For that, it learns a sequence of action-value functions in parallel, where each function is optimized to represent the application of the Bellman operator over the previous function in the sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as each function tracks a moving target. In this work, we modify iterated TD learning by computing the gradients over those moving targets, aiming to build a powerful gradient TD method that competes with semi-gradient methods. Our evaluation reveals that this algorithm, called Gradient Iterated Temporal-Difference learning, has a competitive learning speed against semi-gradient methods across various benchmarks, including Atari games, a result that no prior work on gradient TD methods has demonstrated.

No ratings yet

View paper →

Internal Pluralism and the Limits of Pairwise Comparisons

Bailey Flanigan, Michelle Si

arXiv·2026

Local pairwise comparisons are a standard tool for learning how people want decision rules to work, e.g., in participatory design or alignment. However, their use builds in two strong assumptions: that local comparisons are sufficient evidence about how a person wants an automated decision rule to behave, and that people can always answer those comparisons decisively. We investigate how these assumptions may be compromised under internal pluralism: the idea that an individual evaluates decision rules according to multiple authoritative priorities about how the rule should behave. We provide a formal model of such pluralistic preferences over decision rules, which then lets us identify two distinct failures of forced local pairwise comparison data. First, priorities such as proportionality, egalitarianism, and equal treatment are inherently global: what they imply in one case can depend on what happens elsewhere, so local comparisons may fail to capture them. Second, even when priorities are representable locally, tension between strongly-held priorities can generate internal conflict, producing potentially costly behavioral distortions when comparisons are forced. We then use our model to investigate the alternative -- allowing people to report indecision -- and our findings suggest that doing so can considerably reduce the number of queries needed to learn preferences accurately. We conclude by describing how our model points toward preference-learning methods that elicit these priorities directly, yielding more faithful and interpretable accounts of what people value.

No ratings yet

View paper →

MMPX Style-Preserving Pixel Art Magnification

Morgan McGuire, Mara Gagiu

Journal of Graphics Techniques·2021

We present MMPX, an efficient filter for magnifying pixel art such as 8- and 16-bit era video game sprites, fonts, and screen images by two in each dimension. MMPX preserves art style, attempting to predict what the artist would have produced if working at a larger scale but within the same technical constraints. Pixel art magnification enables displaying classic games and new retro-styled ones on modern screens at runtime, provides high quality scaling and rotation of sprites and raster font glyphs through precomputation at load time, and accelerates content creation workflow. MMPX reconstructs curves, diagonal lines, and sharp corners while preserving the palette, transparency, and single-pixel features. For general pixel art, it can often preserve more as- pects of the original art style than previous magnification filters such as nearest-neighbor, bilinear, HQX, XBR, and EPX. In specific cases and applications, other filters will be better. We recommend EPX and base XBR for content with exclusively rounded corners, and HQX and antialiased XBR for content with large palettes, gradients, and antialiasing. MMPX is fast enough on embedded systems to process typical retro 64k-pixel full screens in less than 0.5 ms on a GPU or CPU. We include open source implementations in C++, JavaScript, and OpenGL ES GLSL for our method and several others.

No ratings yet

View paper →

Lowest Rated Papers

Forecasting With LLMs: Improved Generalization Through Feature Steering

Reimagining Social Relations Through Care and Collective Joy: The Formation of Collectivity Among Volunteers Along the Lines of In/Formality

Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

Forager: a lightweight testbed for continual learning with partial observability in RL

Deep Double Q-learning

Empirical Design in Reinforcement Learning

Pre-training under infinite compute

Efficient Off-Policy Learning for High-Dimensional Action Spaces

Position: Benchmarking is Limited in Reinforcement Learning Research

Gradient Iterated Temporal-Difference Learning

Internal Pluralism and the Limits of Pairwise Comparisons

MMPX Style-Preserving Pixel Art Magnification