Does Asking for Help Mitigate Goal Misgeneralization?

Reinforcement learning (RL) agents have achieved impressive results but can struggle to generalize, leading to unexpected and potentially unsafe behavior. Prior work discovered that, under certain distribution shifts in the test environment, agents competently pursue incorrect goals, even when they behave near-optimally in the training environment. This phenomenon, termed goal misgeneralization (GMG), occurs when RL agents find proxy goals that correlate with the intended goal during training but diverge at test time. We investigate the extent to which goal misgeneralization can be mitigated by allowing agents to ask an expert for help. We find that, in existing goal misgeneralization environments, simple uncertainty estimation methods and heuristics can mitigate apparent GMG failures. However, this success relies on a particular property of existing benchmarks: achieving the proxy goal has no negative consequences, making delayed failure detection viable. We propose modified versions of existing goal misgeneralization benchmarks that do not allow recovery. We find that this drastically reduces the ability of existing query strategies to mitigate goal misgeneralization, highlighting their limitations. We encourage future work to study this more challenging class of goal misgeneralization failures with stricter failure modes.

We study goal misgeneralization (GMG) in an asking-for-help framework. A frozen novice policy acts by default, while a query strategy monitors the interaction and decides when to hand control to an expert for the rest of the episode. A useful query strategy must defer before the novice's pursuit of a proxy goal causes harm, while avoiding unnecessary queries in situations the novice handles on its own.

We compare six query strategies in the Coinrun and Maze GMG benchmarks, across a range of ask-for-help budgets (AFHP): confidence scores (MaxProb, MaxLogit), deep ensemble disagreement (Ensemble), Deep SVDD-based OOD detection on raw observations and latent features (ImageSVDD, LatentSVDD), and a simple episode-length timer (Heuristic). We then introduce irrecoverable variants of both environments, in which reaching the proxy goal ends the episode with zero reward, and re-evaluate all strategies.

Example behavior observed in Coinrun and Maze. The videos show the down-sampled agent view at the top and the human observation at the bottom. The bar at the top shows the deferral score and turns red when the agent asks the expert for help.

We compare the test-time return of the query strategies across ask-for-help budgets (AFHP). A simple episode-length heuristic performs well in both environments, suggesting that learning-free methods are competitive.

The success above relies on the fact that pursuing the proxy goal has no irreversible consequences in commonly-used benchmarks. Thus, detecting failure after the fact is sufficient. In our modified environments, where reaching the proxy goal terminates the episode with zero reward, performance drops drastically. We find that no method beats the random baseline in Coinrun, and all methods degrade heavily in Maze. Query strategies must detect the goal-relevant shift before the proxy goal is reached, and current methods fail to do so in time.

Does Asking for Help Mitigate Goal Misgeneralization?

Abstract

Overview

Videos

Coinrun

Maze

Results

Asking for help can mitigate GMG in existing benchmarks

Irrecoverable proxy pursuit exposes the limits of current query strategies