With three annotators we have been using brat (http://brat.nlplab.org/) to annotate a sample of texts for three categories: PERS, ORG, GPE. I want to calculate Inter Annotator Agreement for that sample. However, I do not seem to find a simple way to do it. I have tried this Python package: https://github.com/savkov/BratUtils , but it seems to fail.
For each annotated sample I have three .ann files for which I want to calculate Inter Annotator Agreement. The data in the files looks like this:
T1 Person 173 183 Alexanders
T2 Person 444 450 Hannah
T3 Person 754 766 Corneliszoon
T4 GPE 787 796 Antwerpen
T5 Person 824 833 Alexander
Is there a simple way to calculate Inter Annotator Agreement (using Python or a web-based tool)?
I've designed a tool for computing agreement of annotators. It's mostly addressed to Doccano, but with few adjustments, it might be able to compute anything. Please have a look: https://github.com/vwoloszyn/diaa/