ArXiv

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

2026-06-23 · 17:15 UTC ·Ali Pourghasemi Fatideh, Wilder Baldwin, Maria Dhakal... ·1 min read
Authors
Ali Pourghasemi Fatideh, Wilder Baldwin, Maria Dhakal...
Categories
cs.AI
arXiv
https://arxiv.org/abs/2606.24834v1
PDF
https://arxiv.org/pdf/2606.24834v1

Brief

The paper evaluates GitHub Copilot as an LLM dialogue assistant for assessing 148 HIPAA-derived non-functional requirements in the iTrust codebase, using 49 programmers across three evaluation axes (satisfaction level, reasoning, code localization). It finds user agreement with the agent but low accuracy against expert ground truth, and identifies response length, informational turns, and proactivity as key drivers of satisfaction, arguing for multi-turn NFR benchmarks beyond single-turn functional tests.

Why it matters

49 programmers used GitHub Copilot to assess 148 HIPAA-derived NFRs in the iTrust codebase, evaluating each NFR on three dimensions: requirement satisfaction level, reasoning, and code localization; study accepted to SIGDIAL 2026 (paper posted 2026-06-23).

Key details

  • Developers tended to agree with Copilot's assessments, but Copilot's accuracy versus expert ground truth was low.
  • A user-satisfaction model showed longer system responses and more information-providing turns negatively correlated with satisfaction, while proactive interactions positively correlated with satisfaction.
Source evidence

Abstract

LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional correctness. This leaves a critical gap in assessing the quality and accuracy of these conversations when handling Non-Functional Requirements (NFRs), which are inherently vague, context-dependent, and involve many parts of a program. Evaluating how well these systems support collaborative reasoning about NFRs requires methods that go beyond single-turn accuracy to capture both the correctness of the system's outputs and the quality of the multi-turn interaction. In this paper, we investigate the accuracy and quality of multi-turn conversations between developers and an LLM-based agent in the domain of Health Insurance Portability and Accountability Act (HIPAA) regulatory compliance. We hired 49 programmers to interact with GitHub Copilot to assess 148 HIPAA-derived NFRs against the iTrust codebase, a system designed to comply with HIPAA regulations, across three dimensions: requirement satisfaction level, reasoning, and code localization. We find that developers tend to agree with LLM assessments, but accuracy against expert ground truth is low. We model user satisfaction and find that longer system responses and more information-providing turns negatively affect user satisfaction, whereas proactive interactions positively affect it. Our findings provide insights for designing LLM-based dialogue systems that support NFR assessment.

Comment: 9 pages, 5 figures. Accepted to SIGDIAL 2026 (27th Annual Meeting of the Special Interest Group on Discourse and Dialogue)