Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data

2026-07-29 · 15:47 UTC ·Lingyang Zeng, Guangze Chen, Kaichen Yu... ·1 min read

Brief

Setoka is a benchmark for memory-augmented personalized agents that defines four hierarchical user-understanding levels (semantic memory, episodic memory, behavior pattern, personality trait) and uses a psychometrics-based pipeline to synthesize heterogeneous, privacy-preserving user data and queries. The authors test 3 LMs with 5 memory systems on 10 synthetic users and show strong performance on explicit semantic retrieval but declining accuracy for episodic recall and much larger failures on behavior-pattern and personality-trait tasks, highlighting the need for memory designs supporting cross-source integration and long-term abstraction.

Why it matters

Setoka is a new benchmark (published 2026-07-29) that evaluates hierarchical user understanding across four levels—semantic memory, episodic memory, behavior pattern, and personality trait—using a psychometrics-based pipeline to synthesize diverse, coherent heterogeneous user data and queries at scale.

Key details

The authors evaluated 3 language models combined with 5 memory systems on 10 synthetic users, finding that existing systems perform well on semantic-memory (explicit fact) retrieval but show notable degradation on episodic-memory tasks and even larger drops for behavior-pattern and personality-trait tasks that require cross-source integration over time.

Results indicate current memory-augmented personalized agents rely on simple fact retrieval and motivate designing memory mechanisms for cross-source integration and abstraction over long-term user behavior.

Abstract

Personalized agents are increasingly applied to assist users across a wide range of tasks. Effective personalized assistance requires not only retrieving explicit facts from past interactions stored in agent memory, but also inferring abstract personal characteristics. However, existing memory benchmarks primarily evaluate whether an agent can retrieve information explicitly stated in conversational histories, failing to provide an effective assessment of deeper user understanding. In this work, we propose Setoka, a benchmark for evaluating memory-augmented personalized agents with hierarchical user understanding from heterogeneous data. Grounded in theories from cognitive and personality psychology, Setoka defines four levels of user understanding, i.e., semantic memory, episodic memory, behavior pattern, and personality trait. Moreover, to enable realistic yet privacy-preserving evaluation, we design a psychometrics-based pipeline that synthesizes diverse, coherent heterogeneous user data and queries at scale. Finally, we leverage Setoka to evaluate 3 language models combined with 5 memory systems for 10 synthetic users. Our comprehensive evaluation reveals that while existing systems perform well on semantic memory retrieval, their performance declines on episodic memory. Moreover, when dealing with behavior pattern and personality trait understanding tasks that require integrating heterogeneous and fragmented information dispersed over time, performance declines even further. These findings demonstrate that user understanding cannot be handled by simple fact retrieval, motivating the design of memory mechanisms for cross-source integration and abstraction over long-term user behavior.