Whitespace is missing in my PetitParser map Function

Question

I tried to parse normal text with function calls in it.

class EmmaGrammarDefinition extends GrammarDefinition {
  const EmmaGrammarDefinition();

  Parser start() => ref(value).plus().end();

  Parser token(Object source, [String name]) {
    if (source is String) {
      return source.toParser(message: 'Expected ${name ?? source}').trim();
    } else if (source is Parser) {
      ArgumentError.checkNotNull(name, 'name');
      return source.flatten('Expected $name').trim();
    } else {
      throw ArgumentError('Unknown token type: $source.');
    }
  }

  Parser word() =>
      ref(wordToken) &
      (whitespace().plus() | ref(token, '.') | whitespace()).optional();

  Parser call() =>
      ref(wordToken) & ref(token, '(') & ref(value) & ref(token, ')');

  Parser array() =>
      ref(token, '[') & ref(elements).optional() & ref(token, ']');

  Parser elements() =>
      ref(value).separatedBy(ref(token, ','), includeSeparators: false);

  Parser members() =>
      ref(pair).separatedBy(ref(token, ','), includeSeparators: false);

  Parser object() =>
      ref(token, '{') & ref(members).optional() & ref(token, '}');

  Parser pair() => (ref(wordToken)| ref(stringToken)) & ref(token, ':') & ref(value);

  Parser value() =>
      ref(call) |
      ref(word) |
      ref(stringToken) |
      ref(numberToken) |
      ref(object) |
      ref(array) |
      ref(trueToken) |
      ref(falseToken) |
      ref(nullToken);

  Parser trueToken() => ref(token, 'true');

  Parser falseToken() => ref(token, 'false');

  Parser nullToken() => ref(token, 'null');

  Parser stringToken() => ref(token, ref(stringPrimitive), 'string');

  Parser numberToken() => ref(token, ref(numberPrimitive), 'number');

  Parser wordToken() => ref(token, ref(wordPrimitive), 'word');

  Parser characterPrimitive() =>
      ref(characterNormal) | ref(characterEscape) | ref(characterUnicode);

  Parser characterNormal() => pattern('^"\\');

  Parser characterEscape() => char('\\') & pattern(jsonEscapeChars.keys.join());

  Parser characterUnicode() => string('\\u') & pattern('0-9A-Fa-f').times(4);

  Parser numberPrimitive() =>
      char('-').optional() &
      char('0').or(digit().plus()) &
      char('.').seq(digit().plus()).optional() &
      pattern('eE')
          .seq(pattern('-+').optional())
          .seq(digit().plus())
          .optional();

  Parser wordPrimitive() => letter().plus() & digit().plus().optional();

  Parser stringPrimitive() =>
      char('"') & ref(characterPrimitive).star() & char('"');
}

It´s close but I have some problems with the normal text part. When I want to map these the point is there but the whitspace is missing.

Parser word() => super.word().map((each) => WordValueWidget(each));

when I parse "Hello World." the each is first ["Hello", null] then ["World", "."] i would like it to be first ["Hello", " "] then ["World", "."].

Thanks for help and sorry for my bad english :)

Answer 1

I suspect that wordToken already consumes all whitespace after the word characters (due to the .trim() operations in your token definition). Then none of the choices in whitespace().plus() | ref(token, '.') | whitespace() match, and .optional() returns null.

That said, it is very hard to tell from looking at a large and complicated grammar like the one you posted. Ideally you would try to break down the problem into smaller and more manageable pieces, that can then be tested and fixed independently. A minimally reproducible example would make it easier for me to provide a directly applicable solution.