How to test a token in JavaCC

Question

I want to create a token that accepts only even numbers, and I need to test it in javacc, I'm just starting to use it and I can't think of any way to test this token, can anyone help me, thanks!

TOKEN: //I want this token to only accept even numbers
{
  <TOKENPARANUMEROSPARES: (["0"-"9"])* ("0"|"2"|"4"|"6"|"8") > {System.out.println("Soy un numero par");}
}

void llamarNumeroPar (): 
{}
{
//I want my program to be able to launch a message from my main with the message shown below
  < TOKENPARANUMEROSPARES > {System.out.println ("soy un numeroPar");}
}

To print something, what do you recommend? Do I do it from the main or what could I correct in this part of my code?

Answer 1

Let's say you have the following parser.jj file:

options {
    STATIC = false;
    IGNORE_CASE = false;
}

PARSER_BEGIN(Parser)
package test14;
public class Parser {
}
PARSER_END(Parser)

TOKEN:
{
  <EVEN: (["0"-"9"])* ("0"|"2"|"4"|"6"|"8") >
}

void llamarNumeroPar (): 
{}
{
  < EVEN > {System.out.println ("soy un numeroPar");}
}

You can now write a main class:

package test14;

import java.io.StringReader;

public class Test14 {
    public static void main(String[] args) {
        Parser parser = new Parser(new StringReader("1234567"));
        Token token = parser.getNextToken();
        if (token.kind == Parser.EVEN) {
            System.out.println("Even");
        } else {
            System.out.println("Something else");
        }
    }
}

And... surprise: the output is "Even". This is because it has read "123456" which is indeed an even number.

Now add a second token ODD:

TOKEN:
{
  <EVEN: (["0"-"9"])* ("0"|"2"|"4"|"6"|"8") >
|  <ODD: (["0"-"9"])* ("1"|"3"|"5"|"7"|"9") >
}

And modify the main class as follows:

    if (token.kind == Parser.EVEN) {
        System.out.println("Even: " + token.image);
    } else if (token.kind == Parser.ODD) {
        System.out.println("Odd: " + token.image);
    } else {
        System.out.println("Something else: " + token.image);
    }

This time, the output is as expected:

Odd: 1234567

Consider defining a token NUMBER and dealing with its parity in the Java code.

Answer 2

Maurice's answer is excellent. I'm going to come at it from a slightly different angle by answering a slightly broader question.

I don't think it makes a lot of sense to test one token definition at a time. This is because the tricky part of writing token managers is usually not the individual regular expressions, but rather the interactions between them, especially when there are multiple states. So the broader question is how to test a token manager as a whole.

I think something like the following could be quite useful.

import your.favourite.test.framework ;

class TokenTests implements XYZConstants {

    private final int TME = -99 ; // -99 because this should not be already in use as a token kind.

    private void doOneTokenTest( String input, int[] expectation ) {
        XYZ parser = new XYZ(new StringReader(input));
        for( int k = 0 ; k < expectation.length; ++k ) {
            int state = parser.token_source.curLexState ;
            try {
                Token token = parser.getNextToken();
                checkThat( expectation[k] != TME,
                       "Expected a TokenMgrErr, but got a" +
                       tokenImage[ token.kind ] + " at token " +
                       k + " of input <<" + input + ">>" +
                       " state is " + state ) ;
                checkThat( token.kind == expectation[k],
                       "Expected a " +
                       tokenImage[ expectation[k] ] +
                       " but got a " +
                       tokenImage[ token.kind ] + " at token " +
                       k + " of input <<" + input + ">>" +
                       " state is " + state ) ;
            } catch( TokenMgrErr e ) {
                checkThat( expectation[k] == TME,
                       "Expected a " +
                       tokenImage[ expectation[k] ] +
                       " but got a TokenMgrErr at token " +
                       k + " of input <<" + input + ">>" +
                       " state is " + state ) ; } } }

So now you can create and run a bunch of tests

    @Test
    public void testTokenMatching() {
        record Pair( String input, int[] expectation ) ;
        Pair[] tests = {
            new Pair( "1234", int[]{ EVEN, EOF } ),
            new Pair( "1233", int[]{ ODD, EOF } ),
            new Pair( "". int[] { EOF },
            new Pair( "123.3", int[]{ ODD, TME } } ;
        for( Pair t : tests ) {
            doOneTokenTest( t.input(), t.expectation() ) ; } }