Converting from XML annotations to BRAT format

Question

I have an annotated data set in XML format: see example below

Treatment of <annotation cui="C0267055">Erosive Esophagitis</annotation> in patients

where the tagged words are in XML tags as shown. I need to get it into BRAT format, such as:

T1    annotation 14 33    Erosive Esophagitis

More examples can be found in http://brat.nlplab.org/standoff.html

I can extract the annotations using regular expressions in Python, but I am unsure of how to get it into the proper BRAT format. Is there a tool for this possibly?

Answer 1

In case someone still needs an answer to this question, here is a solution.

Let's say an XML file sample.xml has the following structure:

<root>
<p n='1'>Hi, my name is <fname>Mickey</fname> <lname>Mouse</lname>, and what about yourself?</p>
<p n='2'>Nice meeting you, <fname>Mickey</fname>! I am <fname>Minnie</lname>!</p>
</root>

Here is a Python solution:

# leave empty if there are no tags that should not be interpreted as named entities; or add more
ignoretags = ['root', 'p']

# dictionary, in case some named entities have to be mapped; or just a list of tags that represent NEs
replacetags = {
    "fname": "PERS",
    "lname": "PERS"
}

# read content
content = open('sample.xml', encoding='utf-8').read()

# output files for BRAT: txt and annotations
f_txt = open('sample.txt', 'w')
f_ann = open('sample.ann', 'w')

# from txt file remove NE tags
clean_content = content
for replacetag in replacetags:
    clean_content = clean_content.replace('<{}>'.format(replacetag), '')
    clean_content = clean_content.replace('</{}>'.format(replacetag), '')

# write content to file
f_txt.write(clean_content)

# char by char
n = len(content)
i = - 1

# token id
tid = 0
# other - for output
start = -1
end = - 1
token = None
tag = None

# let's start parsing! character by character
skipped_chars = 0
number_of_tags = 0

token_buffer = ''
while i < n - 1:

    i += 1
    c = content[i]

    # beginning of an entity
    if c == '<':

    # probably the most important part: always track the count of skipped characters
        start = i - skipped_chars

        # get name of the entity
        tag_buffer = ''
        i += 1
        while content[i] != '>':
            tag_buffer += content[i]
            i += 1
        tag = tag_buffer

    # skip tags that are not NEs
        if tag not in replacetags:
            continue

        # get entity itself
        ent_buffer = ''
        i += 1
        while content[i] != '<':
            ent_buffer += content[i]
            i += 1
        token = ent_buffer

    # determine positions
        end = start + len(token)
        number_of_tags += 1

    # <fname></fname> i.e. 1 + len('fname') + 1 + 1 + 1 + len('fname') + 1
        skipped_chars += 1 + len(tag) + 1 + 1 + 1 + len(tag) + 1
        tid += 1

    # write annotation
        f_ann.write('T{}\t{} {} {}\t{}\n'.format(tid, replacetags[tag], start, end, token))

        # navigate to the end of the entity span, e.g. go behind <fname>...</fname>
        i += 1 + len(tag) + 1

Content of sample.txt

<root>
<p n='1'>Hi, my name is Mickey Mouse, and what about yourself?</p>
<p n='2'>Nice meeting you, Mickey! I am Minnie!</p>
</root>

Content of sample.ann:

T1  PERS 31 37  Mickey
T2  PERS 38 43  Mouse
T3  PERS 101 107    Mickey
T4  PERS 114 120    Minnie

And visually in BRAT:

A minor tweak would be needed in case of attributes (I added another key 'att' in replacetags dictionary, i.e. a pair would be then "fname": {"tag": "PERS", "att": "value of attribute"} and then an additional line would be written in case of tag that has attribute.

Hope someone will find this helpful!