“This answer is better” is a useful opening to a discussion, but an incomplete description of a criterion. Better for which question? In what way? An answer that works for one task may be inconvenient for another.
For Tone-of-Voice evaluation, I designed task-specific rubrics and conducted blind, side-by-side comparisons of AI answers. The research design involved 200 paired answers and 64 users.
Start with the criterion
A rubric makes evaluation more explicit. It provides a language for discussing the quality of an answer, rather than collapsing everything into a general impression.
Connecting criteria to the type of task matters here. Brevity, for example, can be useful in one situation and leave out a necessary explanation in another. A rubric needs to make that distinction explicit, so an assessment relates to the particular task.
My contribution was to develop the rubrics and conduct the comparisons. Tone became part of the research design: what exactly should be evaluated, and how should two responses to a task be compared?
Why paired and blind
A paired comparison gives someone a concrete reference point: the other answer. Instead of judging from memory or against an imagined ideal, they compare two alternatives.
In a blind format, the system’s name is hidden: the person has two answers to compare. The comparison is framed around the content of that pair.
What the study describes
The 200 pairs and 64 users describe this particular research design. They are not a model ranking or a measure of subsequent product improvement.
The substantive work is connecting criteria, tasks and a way of comparing responses. A preference becomes more useful when its scope is clear. Research into trust in AI answers sits alongside this topic: the qualities of the text belong in the same discussion as how someone checks it and continues their work.