Separate retrieval from answer quality
First ask whether the system found the right source. Then ask whether the answer accurately reflects it. A fluent answer cannot compensate for retrieving an outdated or irrelevant document.
Evaluate citation support
A citation is useful only when the referenced material supports the associated claim. Review examples where an answer combines several sources or where different documents contradict each other.
Assign source ownership
Knowledge changes. Identify who maintains each source, how updates reach the retrieval system and how withdrawn material is removed. Without this ownership, answer quality can deteriorate even when the model stays the same.
Test permissions as a workflow
Use questions from different user roles. Check that restricted material is excluded before retrieval and that an answer does not expose information through a summary or citation the user cannot access.
Reward appropriate uncertainty
Include questions whose answers are absent from the approved corpus. The expected behaviour may be refusal, clarification or referral to a person. A system that always answers confidently is not necessarily the most useful one.