ArXiv

SocietyBench: Forecasting Counterfactual Social-World Evolution

2026-08-04 · 17:59 UTC ·Zhenran Wang, Zhonghan Bian, Jinsong Li... ·1 min read
Authors
Zhenran Wang, Zhonghan Bian, Jinsong Li...
Categories
cs.CL
arXiv
https://arxiv.org/abs/2608.04009v1
PDF
https://arxiv.org/pdf/2608.04009v1

Brief

SocietyBench benchmarks models' ability to forecast social-world evolution by building anonymized, counterfactual timelines from web news and social media and turning each cutoff into scored prediction tasks. The suite spans five events, 125 prediction points in Chinese and English, and scores calibration and temporal accuracy separately; results show substantial room for improvement (top LLM 75.0/100, baseline 50) and large per-event variability.

Why it matters

SocietyBench constructs counterfactual, date-indexed social timelines by replacing all named entities and shifting every event date by a per-event constant, then converts every cutoff into audited forecasting questions across five heterogeneous events and 125 prediction points.

Key details

  • On Chinese and English editions, six leading LLMs plus three agent frameworks were evaluated on two orthogonal 100-point axes (probability calibration and temporal accuracy); the best LLM scored 75.0/100 versus a trivial 50 baseline, per-event axis gaps reached 21.4 points, and the three agent frameworks failed to improve the base model while two model-free heuristics trailed all LLMs.
Source evidence

Abstract

Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a browser, operate a GUI. A complementary social ability, namely how well a model understands and forecasts the way real social events unfold, has barely been measured. We introduce SocietyBench, an end-to-end benchmark that takes a one-line event topic, collects Web news and social-media posts across five platforms, distills them into a date-indexed timeline that keeps factual events and a public-opinion layer separate, and then turns every cutoff date on that timeline into an audited bank of forecasting questions. Questions are scored on two orthogonal 100-point axes: probability calibration and temporal accuracy. Before any model sees a timeline, a three-phase procedure replaces every named entity and shifts every date by a per-event constant, turning a real arc into a counterfactual social world -- structurally identical to what happened, but stripped of the surface labels a model could match against pre-training memory. On five heterogeneous events and 125 prediction points in Chinese and English editions, the strongest of six frontier LLMs reaches only 75.0 out of 100, against a trivial anchor of 50. The two axes come apart: a model can be calibration-strong but time-weak, or the reverse. Three agent frameworks built on a shared base model fail to improve on that base, and two model-free heuristics trail every LLM. Per-event gaps reach 21.4 points on a single axis, which is our main argument for evaluating on several events rather than one. All anonymized timelines, question banks, ground truth, and scoring code are released.

Comment: Project page: https://co-minder.github.io/SocietyBench