Why does my caret jump around?

Question

I'm using RSyntaxTextArea together with a custom TokenMaker. The TokerMaker seems to be working well in the sense that the highlighting is working as expected. For example, in the line "class Fibonacci", class is highlighted blue, since it's a reserved keyword, and Fibonacci is simple black, since it's an identifier.

However the caret navigation is not working correctly anymore, both when using the mouse and the keyboard. For example: when I use right-arrow to move the caret
from "clas|s Fibonacci" to "class| Fibonacci" (where | indicates the caret)
then instead it goes to "class Fibon|acci"

Why does the caret jump around like this?

PS: My TokenMaker generates the following calls to addToken for this line:

addToken(text, 0, 4, TokenTypes.RESERVED_WORD, 0);
addToken(text, 5, 5, TokenTypes.WHITESPACE, 0);
addToken(text, 6, 14, TokenTypes.IDENTIFIER, 0);
addNullToken();

Answer 1

I found the problem, the fourth argument of my call to addToken was incorrect. The correct way to calculate the fourth argument is as follows:

override def getTokenList(text: Segment, initialTokenType: Int, startOffset: Int): Token = {
    //some code that defines start, end and tokenType
    val documentOffset = start - text.offset + startOffset;
    addToken(text, start, end, tokenType, documentOffset);
}