I used the irr package from R to calculate a Fleiss kappa statistic for 263 raters that judged 7 photos (scale 1 to 7). kappam.fleiss(db) delivered the kappa statistic (0.554; z=666) and the p-value (0), but unfortunately there is no confidence interval for the kappa statistic included. Can anybody help me out on how I can get the confidence interval ?
thx
Addition of example: row names/ rater.1 / rater.2 / rater.3 / rater.4 / rater.5 / ..../ rater.263 photo 1 / 6 / 6 / 6 / 6 / 7 / ... / 5 photo 2 / 1 / 2 / 1 / 1 / 1 / ... / 2 photo 3 / 5 / 5 / 5 / 5 / 6 / ... / 6 photo 4 / 3 / 1 / 3 / 3 / 3 / ... / 1 photo 5 / 2 / 3 / 2 / 2 / 2 / ... / 3 photo 6 / 4 / 4 / 4 / 4 / 4 / ... / 4 photo 7 / 7 / 7 / 7 / 7 / 5 / ... / 7
The difference between the kappam.fleiss function and the concordance function is that the first is for detailed raters, and the second is for summary data. look at the following example from the Wikipedia page: