How to make same lexer for two parser grammars

Question

I am writing a parser for a language. After implementation I have found that there are so many parser rules. So, I split the grammars. I have added the lexer rules in both the grammars. But, now I have found that the lexer is generated from the first grammar and they are not matched in the other grammar and hence no viable alt exception.

Can anyone guide me how to use the same lexer rules in both the grammars?

Answer 1

You can use the same lexer, you can set the lexer file with the starting line, assuming 'SomeLanguage' as your language:

lexer grammar SomeLanguage;

On your paser file(s) you should start with:

parser grammar SomeLanguage;
options {
    tokenVocab=SomeLanguage;
}

where 'options' makes a reference to the lexer for the grammar.

To compile let's assume the lexer is on a folder called 'the_lexer' and your parser in folder 'the_parser':

java org.antlr.v4.Tool the_lexer\SomeLanguage.g
java org.antlr.v4.Tool -lib the_lexer the_parser\SomeLanguage.g

Answer 2

You can use the same lexer rules in all of your grammars by using the rule import.

Suppose you have a lexer rule grammar like this

   lexer grammar MainLexer;

   Protocol
   : 'protocol'
   ;

   Start
   : 'start'
   ;

   Identifier
   : ('a'..'z' | 'A'..'Z')+
   ;

And the task is to reuse this lexer in every grammar.

Write an dummy Lexer grammar (In the second grammar) which imports the main Lexer like this

    lexer grammar Dummy1Lexer;

    import MainLexer;        

    Dummy1
    : 'dummy1'
    ;

However you have to make the main Lexer grammar available in the 2nd grammar project .