Stanford NER is not properly extracting percentages

Question

I'm trying to extract percentages using Stanford NER. But it is not extracting percentage properly.

inp_str = 'total revenue received was one hundred and twenty five percent 125% for last financial year'
split_inp_str = inp_str.split()
st = StanfordNERTagger('english.muc.7class.distsim.crf.ser.gz')
print(st.tag(split_inp_str))

This gives following output

[('total', 'O'), ('revenue', 'O'), ('received', 'O'), ('was', 'O'), ('one', 'O'), ('hundred', 'O'), ('and', 'O'), ('twenty', 'O'), ('five', 'PERCENT'), ('percent', 'PERCENT'), ('125%', 'O'), ('for', 'O'), ('last', 'O'), ('financial', 'O'), ('year', 'O')]

Why is it not extracting 125% or one hundred and twenty five percent?

Answer 1

You need to tokenize the sentence rather than split(). Try the following code.

from nltk import word_tokenize

inp_str = 'total revenue received was one hundred and twenty five percent 125% for last financial year'
split_inp_str = word_tokenize(inp_str)
st = StanfordNERTagger('english.muc.7class.distsim.crf.ser.gz')
print(st.tag(split_inp_str))