ArXiv

Know It, Act on It: Investigating Memory Utilization in LLM Personalization

2026-07-31 · 13:57 UTC ·Zhaoxin Feng, Jianfei Ma, Emmanuele Chersoni ·1 min read
Authors
Zhaoxin Feng, Jianfei Ma, Emmanuele Chersoni
Categories
cs.CL
arXiv
https://arxiv.org/abs/2607.29433v1
PDF
https://arxiv.org/pdf/2607.29433v1

Brief

The paper studies memory utilization in LLM personalization by introducing a paired Know-and-Act testing paradigm that disentangles whether agents forget user preferences or remember but fail to use them. Based on the abstract, experiments across 16 systems and five memory architectures with 1,000 preferences at three expression strengths reveal a large gap: agents often pass recall tests yet do not reflect preferences in behavior, with particularly poor utilization for health and therapy–related preferences.

Why it matters

Introduces a decoupled evaluation paradigm using paired Know and Act tests to separate recall from utilization of a user preference.

Key details

  • Large-scale evaluation (1,000 preferences at three expression strengths) across 16 systems and five memory architectures shows a substantial Know–Act gap: models often recall preferences but fail to act on them.
  • Memory architectures reduce the gap but utilization remains especially weak for health and therapy–related preferences; paper by Zhaoxin Feng, Jianfei Ma, Emmanuele Chersoni (arXiv:2607.29433v1), published 2026-07-31.
Source evidence

Abstract

As large language model (LLM) agents evolve into personalized companions, memory has emerged as a core capability. However, LLMs face a knowledge utilization problem: they may fail to act on relevant user preferences even when they are fully present in context. When an agent fails to tailor its response in a context where previously shared user preferences should matter, it is unclear whether the model failed to remember that information or remembered it but failed to use it. To isolate this breakdown, we introduce a decoupled evaluation paradigm that administers paired Know and Act tests to the same user preference. We conduct large-scale experiments across 16 systems and five memory architectures, evaluating 1,000 preferences embedded at three levels of expression strength. Our results show a large gap between Know and Act outcomes: agents often pass the recall test for a user preference but fail to reflect that same preference in the paired behavioral scenario. While memory architectures reduce this gap, utilization remains especially weak for health and therapy-related preferences, where failures to act carry the greatest real-world stakes.