NLP tokenizer that handles missing white spaces

1k Views Asked by At

Among the open source NLP libraries, is there one whose tokenizer handles missing white spaces? For instance, the phrase "this tokenizer isgreat" would give [this, tokenizer, is, great] instead of [this, tokenizer, isgreat].

0

There are 0 best solutions below