I'm running a Fleiss kappa and the test will not run when there is perfect agreement between the participants. I have had participants evaluate 4 conditions in a yes/no type question on whether the condition had an object present (yes) or not (no). Therefore, it was fairly easy to correctly judge/answer the question. As a consequence there is perfect agreement in 3 conditions (the correct answer was yes), whereas there is not in 1 condition (the correct answer was no). I have ran Fleiss kappa with all conditions together and get a kappa = 0.723.

Now I want to run one Fleiss kappa test for the 3 conditions only (I expect perfect agreement here), and one test the condition for the last condition. Respectively, I get the following error messages:

Warnings

All ratings are the same.

Execution of this command stops.

Warnings

There are too few complete subjects.

Execution of this command stops.

What I can I do to fix this issue?

Hi Henrikke. I don't think the issue is perfect agreement per se. The data are expected to be structured with rated objects as cases and ratings from different raters as columns. If all raters rate each case identically, but there is some variation in the ratings over cases, then you can have complete agreement and get results indicating perfect agreement.

The message for the first analysis that's not running indicates that the ratings are all the same, meaning no variability among the ratings at all. In a situation like this, you'd get 0/0 in attempting to calculate the Fleiss kappa statistic, because while all ratings are in agreement for each subject, with only a single observed rating category, the expected proportion of agreements would be 1. Thus the procedure stops with the issued warning.

The second message indicates that there are fewer than two complete cases available after deletion of cases with missing values. I'm not sure exactly how that happened.