ArXiv

Same Formulas, Different Semantics: Do Language Models Follow Modal Logic Specifications?

2026-08-05 · 17:33 UTC ·Réemi Andrieu, Damien Sileo ·1 min read
Authors
Réemi Andrieu, Damien Sileo
Categories
cs.CL
arXiv
https://arxiv.org/abs/2608.05097v1
PDF
https://arxiv.org/pdf/2608.05097v1

Brief

The paper evaluates whether large language models follow specified modal semantics by generating paired modal formulas (same premises/conjecture, different frames/domains) and verifying opposite labels with automated reasoning. On a balanced core, most models fail the semantics test under direct prompting, but inference mode matters greatly — e.g., DeepSeek V4 Flash jumps from 4.4% to 88.1%. The authors release code, data, oracle artifacts, and countermodels.

Why it matters

The authors (Réemi Andrieu, Damien Sileo; arXiv 2026-08-05) construct paired modal problems with identical premises and conjectures but different frame or domain conditions; automated theorem proving verifies that the pairs receive opposite truth labels under the two semantics.

Key details

  • On a balanced core dataset that prevents the semantic condition alone revealing the answer, four of five recent LLMs perform below a condition-only baseline under direct prompting.
  • Switching inference mode dramatically changes behavior: enabling reasoning mode raises DeepSeek V4 Flash accuracy on the same prompts from 4.4% to 88.1%. Code and artifacts are released (GitHub and Hugging Face).
Source evidence

Abstract

Reasoning about necessity and possibility depends on assumptions about accessibility between worlds and about which objects exist at each one. The same inference may therefore hold under one modal system and fail under another. Evaluating language models on such problems requires testing whether their judgments follow the stated semantics rather than a familiar logic. We construct paired modal problems with identical premises and conjecture but different frame or domain conditions; automated reasoning verifies opposite labels. A balanced core prevents the semantic condition alone from revealing the answer. On this core, four of five recent models perform below the condition-only baseline under direct prompting. Yet enabling reasoning mode raises DeepSeek V4 Flash from 4.4% to 88.1% on unchanged prompts. Following stipulated modal semantics thus depends strongly on inference mode as well as model identity. When frame conditions are omitted, models often agree but fit different familiar logics best. We release the formulas, oracle artifacts, countermodels, and responses.

Comment: 9 pages. Code: https://github.com/sileod/modal-semantics-reasoning. Data and artifacts: https://huggingface.co/datasets/sileod/modal-semantics-reasoning