boost spirit lexer on a stream does not tokenize "early" enough (too much lookahead?)

Question

I'm using a boost::spirit::lex lexer to tokenize an input stream, using spirit::istream_iterators as described in Using Boost.Spirit.Lex and stream iterators .

My problem is that lex::tokenize does not seem to output a token as "early" (along the input stream) as I think it should. It waits for an extra token (or two?) to be available in whole before I get the preceding token. I suppose this is a lookahead-delay, but I'm not sure why it is necessary, or how to get around it.

Here is an example:

#include <boost/bind.hpp>
#include <boost/spirit/include/lex_lexertl.hpp>
#include <boost/spirit/include/support_istream_iterator.hpp>
namespace spirit = boost::spirit;
namespace lex = spirit::lex;

#include <iostream>
using std::cin;
using std::cout;
using std::endl;

template <typename Lexer>
struct ALexer : lex::lexer<Lexer> {
  ALexer() {
    this->self.add
      ("hello")
      ("goodbye")
      ("[ \\t\\r\\n]+")
    ;
  }
};

struct Emitter {
  typedef bool result_type;   // for boost::bind
  template <typename Token>
  bool operator() (const Token& t) const {
    cout << "token: " << t.value() << endl;
    return true;
  }
};

int main() {
  cin >> std::noskipws;

  // iterate over stream input
  spirit::istream_iterator in_begin(cin);
  spirit::istream_iterator in_end;

  typedef lex::lexertl::token<spirit::istream_iterator> token_type;
  typedef lex::lexertl::lexer<token_type> lexer_type;
  typedef ALexer<lexer_type> tokenizer_type;
  tokenizer_type tokenizer;

  (void) lex::tokenize(in_begin, in_end, tokenizer, boost::bind(Emitter(), _1));

  return 0;
}

Example session: (Note that cin is line-buffered by my OS, so the first "line" becomes available to the lexer at once)

(input)    hello goodbye<NEWLINE>
(output)   token: hello
           token:  
(input)    <EOF>
(output)   token: goodbye
           token: 

I want the lexer to receive the first line, and as soon as it gets to the <NEWLINE>, it should know that a full "goodbye" token has been lexed, and emit it. Instead, it seems to wait for more input before I can get the goodbye.