Python Stanza: Some Tokens don't have a lemma?

111 Views Asked by At

I noticed that the token "sep" when using stanza pipeline ("en") does not have a lemma attribute... enter image description here

import stanza 
nlp_tokenize = stanza.Pipeline('en', processors='tokenize,mwt,pos,lemma,depparse', \
   tokenize_pretokenized = True, use_gpu=True)#, use_gpu=False) 
doc = nlp_tokenize("sep") # the word "sep" has no attribute lemma 
print(doc)

This is the only token I've found, where this problem occured (all the other tokens do have the lemma attribute).

Why is that?

0

There are 0 best solutions below