Using BOOST Tokenizer to display delimiter and to not tokenize a string in quotes

Question

I am using BOOST Tokenizer to break a string into toekn. Basically the tokens will be used to create a compiler for VSL based on c/c++. What i wanted to ask that is it possible that the delimiter defined created using

char_separator<char> sep("; << "); 

be also displayed for example if i use Boost tokenizer on string

string s= "cout<<hello;"

it should make the following tokens

cout
<<
hello
;

Also how can i ensure that it does not convert tokenize something in quotes like

string s= "hello my \"name is\" Hassan"

should be converted to following tokens

hello
my
name is
Hassan

Answer 1

I suggest boost spirit: Live On Coliru

Edit See also http://www.boost.org/doc/libs/1_55_0/libs/spirit/example/qi/compiler_tutorial

#include <boost/spirit/include/qi.hpp>

namespace qi = boost::spirit::qi;

int main(int argc, char** argv)
{
    typedef std::string::const_iterator It;
    std::string const input = "cout<<hello;my \"name is\" Hassan";

    qi::rule<It, std::string()> delimiter = qi::char_("; ") | qi::string("<<");
    qi::rule<It, std::string()> quoted    = '"' >> *~qi::char_('"') > '"';
    qi::rule<It, std::string()> word      = +((quoted | qi::char_) - delimiter);

    std::vector<std::string> tokens;
    if (qi::parse(input.begin(), input.end(), *(word >> delimiter), tokens))
    {
        for(auto& token : tokens)
            std::cout << "'" << token <<  "'\n";
    }
}

Output:

'cout'
'<<'
'hello'
';'
'my'
' '
'name is'
' '
'Hassan'