Designing a Coherence Score (CS) for Structural Evaluation of LLM Outputs

A Structural Audit Framework for Multi-Step Reasoning Integrity Forest Code Labs | 2025 Introduction Evaluation has become the bottleneck in modern LLM systems. We have dramatically improved generation speed, context length, and retrieval quality. But the deeper we push multi-step reasoning, agent orchestration, and long-form outputs, the more a different problem emerges: structural drift. An output can be fluent. It can be factually aligned. It can even cite sources correctly. And still fail to preserve its own constraints. Most existing evaluation methods measure probability, similarity, or correctness. Very few measure whether reasoning remains structurally coherent across steps. This paper introduces a Coherence Score (CS): a lightweight structural audit framework designed to evaluate multi-step reasoning integrity in production pipelines. CS does not replace factual evaluation. It does not claim to solve hallucinations. It measures something narrower — and increasingly critical: C