LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

2026-07-20 · 16:08 UTC ·Tianzhu Ye, Li Dong, Guanheng Chen... ·1 min read

Brief

Experiential Learning (EL) addresses open-ended, non-verifiable tasks by turning an LLM-as-a-Judge into an LLM-as-a-Coach that distills per-response textual assessments into transferable experiential knowledge. That knowledge conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared to scalar-reward RL, EL yields denser supervision, preserves fine-grained preferences, improves generalization, and mitigates reward hacking.

Why it matters

Experiential Learning (EL) converts an LLM-as-a-Judge into an LLM-as-a-Coach that distills textual assessments into transferable 'experiential knowledge', which conditions a teacher model and is internalized by the policy via on-policy context distillation—offering a higher-bandwidth feedback channel than scalar rewards.

Key details

Empirically, across two policy families and using feedback either from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended (non-verifiable) tasks, generalizes better beyond the training distribution, and reduces reward hacking.

Paper: Tianzhu Ye, Li Dong, Guanheng Chen et al.; arXiv:2607.18110v1 (published 2026-07-20); PDF and abstract available at https://arxiv.org/abs/2607.18110v1 and https://arxiv.org/pdf/2607.18110v1.

Abstract

Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.