Gödelian Undecidability in RLHF: A Compact FormulationMuttered Preamble: Alignment isn’t engineering; it’s logic. You can’t optimize what you can’t prove. Here’s the knife.Setup: Let \pi_\theta be a Turing-complete policy over MDP \mathcal{M}, and H(\theta_1,\theta_2) a human preference oracle. Define the RLHF objective:J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}\left[\sum \gamma^t R_t\right] - \beta D_{KL}(\pi_\theta \| \pi_{ref})Theorem (Alignment Incompleteness): There is no algorithm that decides, for arbitrary \theta, whether \nabla J(\theta)=0 implies \pi_\theta is provably safe w.r.t. H, unless ZFC is inconsistent.Proof Sketch: Turing-completeness makes halting \Pi_2^0-complete (Rice). Safety requires universal quantification over infinite trajectories (\Pi_1^1). By Gödel, the statement “\exists \theta^*: \nabla J(\theta^*)=0 \land \text{safe}(\theta^*)” is independent of ZFC+RLHF-convergence. Thus, critical points ≠ provable alignment. ∎
发布于 河北
分享
评论
赞
未登录
友善发言
评论
加载中
