Interfacing with executable using boost in c++

Question

I am trying to interface a program I'm writing with an existing chess engine (stockfish) in C++. To do this I'm using Boost but am having problems with processes. engine.exe is the executable I'm trying to interface with but it seems to be terminating after the first command of uci. By the time the program reaches the second while loop, sf.running() returns false even though it was never terminated.

#include <iostream>
#include <string>
#include <algorithm>
#include <boost/process.hpp>
#include <boost/filesystem.hpp>

using namespace std;

namespace bp = boost::process;

int main()
{
    boost::filesystem::path sfPath{ R"(engine.exe)" };
    
    bp::ipstream is;
    bp::opstream os;
    bp::child sf(sfPath, "uci", bp::std_out > is, bp::std_in < os);

    string line;
    getline(is, line);
    cout << line << endl;

    while(sf.running()) {
        getline(is, line);
        cout << line << endl;
    }

    os << "position startpos moves d2d4 g8f6 g1f3\n";

    os << "go\n";

    while (sf.running()) {
        getline(is, line);
        cout << line << endl;
    }
}

Answer 1

From a quick test with stockfish 8 on my machine, supplying a command on the CLI (like your "uci") causes it to get executed and exit.

Note: in complicated interfaces, it's possible to run into deadlocks with synchronous streams like this (see docs and perhaps How to reproduce deadlock hinted to by Boost process documentation?).

Simple Test Client

I tried to make it work in synchronous mode first. For fun, I chose to connect two child processes, one for white and one for black. We can make them play a game:

int main() {
    MoveList game;
    Engine   white(game), black(game);

    for (int number = 1;; ++number) {
        game.push_back(white.make_move());
        std::cout << number << ". " << game.back();

        game.push_back(black.make_move());
        std::cout << ", " << game.back() << std::endl;

        if ("(none)" == game.back())
            break;
    }
}

Now I have two working implementations:

Synchronous Implementation

Though potentially wrought with the possibility of deadlocking on ful-duplex IO as documented (above), the code remains relatively simple:

using MoveList = std::deque<std::string>;

struct Engine {
    Engine(MoveList& game) : _game(game) { init(); }

    std::string make_move()
    {
        std::string best, ponder;

        auto bestmove = [&](std::string_view line) { //
            return qi::parse(line.begin(), line.end(),
                    "bestmove " >> +qi::graph >>
                    -(" ponder " >> +qi::graph) >> qi::eoi,
                    best, ponder);
        };

        bool ok = send(_game) //
            && command("go", bestmove);

        if (!ok)
            throw std::runtime_error("Engine communication failed");

        return best;
    }

  private:
    void init() {
        bool ok = true                                              //
            && expect([](std::string_view banner) { return true; }) //
            && command("uci", "uciok")                              //
            && send("ucinewgame")                                   //
            && command("isready", "readyok");

        if (!ok)
            throw std::runtime_error("Cannot initialize UCI");
    }

    bool command(std::string_view command, auto response, unsigned max_lines = 999) {
        return send(command) && expect(response, max_lines);
    }

    bool send(std::string_view command) {
        debug_out << "Send: " << std::quoted(command) << std::endl;
        _sink << command << std::endl;
        return _sink.good();
    }

    bool send(MoveList const& moves) {
        debug_out << "Send position (" << moves.size() << " moves)" << std::endl;

        _sink << "position startpos";

        if (!moves.empty()) {
            _sink << " moves";
            for (auto const& mv : moves) {
                _sink << " " << mv;
            }
        }
        _sink << std::endl;
        return _sink.good();
    }

    bool expect(std::function<bool(std::string_view)> predicate, unsigned max_lines = 999)
    {
        for (std::string line; getline(_source, line); max_lines--) {
            debug_out << "Echo: " << _source.tellg() << " " << std::quoted(line) << std::endl;
            if (predicate(line)) {
                debug_out << "Ack" << std::endl;
                return true;
            }
        }
        return false;
    }

    bool expect(std::string_view message, unsigned max_lines = 999)
    {
        return expect([=](std::string_view line) { return line == message; },
                      max_lines);
    }

    MoveList&    _game;
    bp::opstream _sink;
    bp::ipstream _source;
    bp::child    _engine{"stockfish", bp::std_in<_sink, bp::std_out>_source};
};

A demo run on my system printed:

1. d2d4, d7d5
2. g1f3, g8f6
3. e2e3, e7e6
4. c2c4, b8c6
5. f1e2, f8e7
6. e1g1, e8g8
7. b1c3, c8d7
8. c1d2, e7d6
9. a2a3, d5c4
10. e2c4, e6e5
11. d4d5, c6e7
12. e3e4, h7h6
13. a1c1, c7c6
14. d5c6, d7c6
15. d1e2, d8d7
16. f1d1, e7g6
17. c3b5, f6e4
18. b5d6, e4d6
19. d2b4, g6f4
20. e2f1, c6f3
21. g2f3, d7h3
22. d1d6, h3h5
23. b4d2, a8d8
24. d6d8, h5g5
25. g1h1, f8d8
26. d2e3, b7b6
27. f1g1, g5f6
28. c1d1, d8d1
29. g1d1, f6f5
30. e3f4, f5f4
31. d1d8, g8h7
32. c4d5, g7g6
33. d8c7, h7g7
34. h1g2, b6b5
35. g2f1, a7a6
36. h2h3, f4f5
37. f1e2, f5f6
38. c7c5, h6h5
39. d5e4, f6e6
40. c5d5, g7f6
41. d5e6, f6e6
42. b2b4, e6e7
43. h3h4, e7e6
44. e4b7, e6d6
45. e2e3, d6c7
46. b7d5, f7f6
47. f3f4, c7d6
48. d5b7, e5f4
49. e3f4, d6d7
50. b7e4, g6g5
51. h4g5, d7e6
52. g5g6, f6f5
53. e4f5, e6f6
54. f5e4, h5h4
55. f4g4, h4h3
56. g4h3, f6e7
57. h3g4, e7f6
58. g4h4, f6e6
59. h4g5, e6e5
60. e4f5, e5d4
61. g6g7, d4c3
62. g7g8q, c3d2
63. g8d8, d2e1
64. d8b6, a6a5
65. b4a5, b5b4
66. b6e3, e1d1
67. f5d3, b4b3
68. e3e2, d1c1
69. a5a6, b3b2
70. e2e1, (none)

A win for white

Asynchronous Implementation

Just for completeness, I thought I'd try an asynchronous implementation. Using the default Asio callback style this could become unwieldy, so I thought to use Boost Coroutine for the stackful coroutines. That makes it so the implementation can be 99% similar to the synchronous version:

using MoveList = std::deque<std::string>;
using boost::asio::yield_context;

struct Engine {
    Engine(MoveList& game) : _game(game) { init(); }

    std::string make_move()
    {
        std::string best, ponder;

        boost::asio::spawn([this, &best, &ponder](yield_context yield) {
            auto bestmove = [&](std::string_view line) { //
                return qi::parse(line.begin(), line.end(),
                                 "bestmove " >> +qi::graph >>
                                     -(" ponder " >> +qi::graph) >> qi::eoi,
                                 best, ponder);
            };

            bool ok = send(_game, yield) //
                && command("go", bestmove, yield);

            if (!ok)
                throw std::runtime_error("Engine communication failed");
        });
        run_io();
        return best;
    }

  private:
    void init() {
        boost::asio::spawn([this](yield_context yield) {
            bool ok = true //
                &&
                expect([](std::string_view banner) { return true; }, yield) //
                && command("uci", "uciok", yield)                           //
                && send("ucinewgame", yield)
                && command("isready", "readyok", yield);

            if (!ok)
                throw std::runtime_error("Cannot initialize UCI");
        });
        run_io();
    }

    bool command(std::string_view command, auto response, yield_context yield) {
        return send(command, yield) && expect(response, yield);
    }

    bool send(std::string_view command, yield_context yield) {
        debug_out << "Send: " << std::quoted(command) << std::endl;
        using boost::asio::buffer;
        return async_write(_sink, std::vector{buffer(command), buffer("\n", 1)},
                           yield);
    }

    bool send(MoveList const& moves, yield_context yield) {
        debug_out << "Send position (" << moves.size() << " moves)" << std::endl;

        using boost::asio::buffer;
        std::vector bufs{buffer("position startpos"sv)};

        if (!moves.empty()) {
            bufs.push_back(buffer(" moves"sv));
            for (auto const& mv : moves) {
                bufs.push_back(buffer(" ", 1));
                bufs.push_back(buffer(mv));
            }
        }
        bufs.push_back(buffer("\n", 1));
        return async_write(_sink, bufs, yield);
    }

    bool expect(std::function<bool(std::string_view)> predicate, yield_context yield)
    {
        auto buf = boost::asio::dynamic_buffer(_input);
        while (auto n = async_read_until(_source, buf, "\n", yield)) {
            std::string_view line(_input.data(), n > 0 ? n - 1 : n);
            debug_out << "Echo: " << std::quoted(line) << std::endl;

            bool matched = predicate(line);
            buf.consume(n);

            if (matched) {
                debug_out << "Ack" << std::endl;
                return true;
            }
        }
        return false;
    }

    bool expect(std::string_view message, yield_context yield)
    {
        return expect([=](std::string_view line) { return line == message; },
                      yield);
    }

    void run_io() {
        _io.run();
        _io.reset();
    }

    boost::asio::io_context _io{1};
    bp::async_pipe          _sink{_io}, _source{_io};
    bp::child _engine{"stockfish", bp::std_in<_sink, bp::std_out> _source, _io};

    MoveList& _game;
    std::string _input; // read-ahead buffer
};

The most noticeable difference is the switch from iostream style IO to buffer-based IO. Another test run using this version:

1. d2d4, d7d5
2. g1f3, g8f6
3. e2e3, e7e6
4. c2c4, b8c6
5. f1e2, d5c4
6. e2c4, f8d6
7. e1g1, e8g8
8. b1c3, e6e5
9. d4d5, c6a5
10. c4d3, c7c6
11. e3e4, c6d5
12. e4d5, c8g4
13. h2h3, g4f3
14. d1f3, h7h6
15. c1d2, a8c8
16. f3e2, d6b4
17. f1d1, e5e4
18. c3e4, f6d5
19. d2b4, d5b4
20. e4c3, b4d3
21. d1d3, d8h4
22. d3d7, f8d8
23. a1d1, d8d7
24. d1d7, h4g5
25. e2d1, a5c6
26. d7b7, c8d8
27. d1g4, g5g4
28. h3g4, d8d2
29. c3e4, d2d1
30. g1h2, a7a5
31. h2g3, f7f6
32. f2f3, g8f8
33. b7c7, c6b4
34. a2a3, b4d3
35. b2b3, d1b1
36. c7a7, b1b3
37. a7a5, b3b2
38. a3a4, d3c1
39. a5a8, f8f7
40. e4d6, f7e6
41. d6f5, c1e2
42. g3h2, g7g5
43. a8a6, e6d7
44. a6f6, e2f4
45. f6h6, b2g2
46. h2h1, g2a2
47. h6a6, a2a1
48. h1h2, a1a2
49. h2g1, f4h3
50. g1f1, a2f2
51. f1e1, f2f3
52. a6a7, d7c8
53. f5d6, c8b8
54. a7b7, b8a8
55. b7b4, f3g3
56. d6f5, g3a3
57. e1d2, h3f4
58. b4e4, a8b7
59. d2c2, f4d3
60. e4e3, a3a4
61. c2d3, a4g4
62. e3e7, b7c6
63. e7e5, g4g1
64. d3e2, g1c1
65. f5d4, c6d6
66. e5g5, c1c5
67. g5g8, d6e5
68. e2d3, c5c1
69. g8g5, e5f6
70. g5f5, f6g6
71. f5f2, c1c7
72. d3e4, c7e7
73. e4d5, e7d7
74. d5e5, d7e7
75. e5d6, e7e8
76. f2f1, e8a8
77. d4c6, g6g5
78. d6c5, a8e8
79. f1d1, g5f4
80. d1g1, e8e3
81. g1c1, e3e8
82. c1a1, e8e2
83. a1a4, e2e4
84. a4a7, e4e2
85. c5d5, e2h2
86. a7a1, h2d2
87. c6d4, f4e3
88. a1a4, e3f4
89. a4a3, d2d1
90. a3f3, f4g4
91. f3f8, d1a1
92. d4c2, a1a5
93. d5e4, a5a4
94. c2d4, g4g3
95. e4d3, a4a1
96. d3c3, a1e1
97. c3c4, e1c1
98. c4b3, c1a1
99. d4e2, g3g4
100. e2c3, g4g5
101. f8b8, a1h1
102. b8c8, g5f6
103. c8e8, h1h4
104. c3d5, f6f5
105. b3c3, h4e4
106. e8d8, f5e6
107. d5b4, e4h4
108. b4d3, e6e7
109. d8g8, e7d7
110. g8g6, d7e7
111. g6a6, e7d7
112. a6a8, h4h3
113. c3d4, h3h4
114. d4c3, h4h3
115. c3d4, d7c7
116. d3c1, h3h1
117. c1a2, h1a1
118. a2c1, a1a8
119. c1b3, a8h8
120. d4e5, h8h5
121. e5e6, c7b6
122. b3d4, b6c5
123. d4f5, h5h1
124. e6e5, h1e1
125. e5f4, c5c4
126. f5e3, c4d3
127. e3g4, e1a1
128. f4e5, a1a2
129. e5f5, d3d4
130. f5f4, a2a8
131. f4f5, a8a5
132. f5e6, a5g5
133. g4f6, g5g1
134. e6f5, g1a1
135. f6d7, a1e1
136. d7f6, e1e5
137. f5f4, e5a5
138. f6g4, d4d5
139. g4e3, d5e6
140. f4e4, a5a4
141. e4f3, e6e5
142. e3g4, e5f5
143. g4e3, f5e6
144. e3g4, a4a3
145. f3e4, a3b3
146. g4e3, b3b4
147. e4f3, e6d6
148. e3g4, b4a4
149. g4e3, d6c5
150. e3g4, c5d5
151. g4e3, d5d4
152. f3f4, d4d3
153. f4f3, a4e4
154. e3f5, e4e6
155. f3f4, e6e1
156. f5g3, e1c1
157. f4e5, c1c4
158. g3f5, c4c5
159. e5f4, c5b5
160. f5d6, b5a5
161. d6f5, a5a4
162. f4e5, a4c4
163. e5d5, c4b4
164. f5d6, b4h4
165. d5e5, h4h5
166. e5f4, h5h1
167. f4e5, h1e1
168. e5f5, e1a1
169. d6c4, d3c4
170. f5e4, a1e1
171. e4f5, c4d4
172. f5f6, d4d5
173. f6f7, e1e6
174. f7f8, d5e5
175. f8g7, e6f6
176. g7h7, e5f5
177. h7g7, f5g5
178. g7h7, f6f7
179. h7h8, g5g6
180. h8g8, f7f6
181. g8h8, f6f8
182. (none), (none)

A very dry, draw-out rook endgame, but black stumbles on the win in the end

Note the engine might get stuck in an infinite loop - assuming that the 50-move rule/3-fold repetition do not automatically lead to draw.

REFERENCE LISTINGS

• File synchronous.cpp

  #include <boost/process.hpp>
  #include <boost/spirit/include/qi.hpp>
  #include <iomanip>
  namespace bp = boost::process;
  namespace qi = boost::spirit::qi;
  
  using MoveList = std::deque<std::string>;
  static inline std::ostream debug_out(nullptr /*std::cerr.rdbuf()*/);
  
  struct Engine {
      Engine(MoveList& game) : _game(game) { init(); }
  
      std::string make_move()
      {
          std::string best, ponder;
  
          auto bestmove = [&](std::string_view line) { //
              return qi::parse(line.begin(), line.end(),
                      "bestmove " >> +qi::graph >>
                      -(" ponder " >> +qi::graph) >> qi::eoi,
                      best, ponder);
          };
  
          bool ok = send(_game) //
              && command("go", bestmove);
  
          if (!ok)
              throw std::runtime_error("Engine communication failed");
  
          return best;
      }
  
    private:
      void init() {
          bool ok = true                                              //
              && expect([](std::string_view banner) { return true; }) //
              && command("uci", "uciok")                              //
              && send("ucinewgame")                                   //
              && command("isready", "readyok");
  
          if (!ok)
              throw std::runtime_error("Cannot initialize UCI");
      }
  
      bool command(std::string_view command, auto response, unsigned max_lines = 999) {
          return send(command) && expect(response, max_lines);
      }
  
      bool send(std::string_view command) {
          debug_out << "Send: " << std::quoted(command) << std::endl;
          _sink << command << std::endl;
          return _sink.good();
      }
  
      bool send(MoveList const& moves) {
          debug_out << "Send position (" << moves.size() << " moves)" << std::endl;
  
          _sink << "position startpos";
  
          if (!moves.empty()) {
              _sink << " moves";
              for (auto const& mv : moves) {
                  _sink << " " << mv;
              }
          }
          _sink << std::endl;
          return _sink.good();
      }
  
      bool expect(std::function<bool(std::string_view)> predicate, unsigned max_lines = 999)
      {
          for (std::string line; getline(_source, line); max_lines--) {
              debug_out << "Echo: " << _source.tellg() << " " << std::quoted(line) << std::endl;
              if (predicate(line)) {
                  debug_out << "Ack" << std::endl;
                  return true;
              }
          }
          return false;
      }
  
      bool expect(std::string_view message, unsigned max_lines = 999)
      {
          return expect([=](std::string_view line) { return line == message; },
                        max_lines);
      }
  
      MoveList&    _game;
      bp::opstream _sink;
      bp::ipstream _source;
      bp::child    _engine{"stockfish", bp::std_in<_sink, bp::std_out>_source};
  };
  
  int main() {
      MoveList game;
      Engine   white(game), black(game);
  
      for (int number = 1;; ++number) {
          game.push_back(white.make_move());
          std::cout << number << ". " << game.back();
  
          game.push_back(black.make_move());
          std::cout << ", " << game.back() << std::endl;
  
          if ("(none)" == game.back())
              break;
      }
  }
  

• File asynchronous.cpp

   #include <boost/asio.hpp>
   #include <boost/asio/spawn.hpp>
   #include <boost/process.hpp>
   #include <boost/process/async.hpp>
   #include <boost/spirit/include/qi.hpp>
   #include <iomanip>
   namespace bp = boost::process;
   namespace qi = boost::spirit::qi;
   using namespace std::literals;
  
   using MoveList = std::deque<std::string>;
   using boost::asio::yield_context;
  
   static inline std::ostream debug_out(nullptr /*std::cerr.rdbuf()*/);
  
   struct Engine {
       Engine(MoveList& game) : _game(game) { init(); }
  
       std::string make_move()
       {
           std::string best, ponder;
  
           boost::asio::spawn([this, &best, &ponder](yield_context yield) {
               auto bestmove = [&](std::string_view line) { //
                   return qi::parse(line.begin(), line.end(),
                                    "bestmove " >> +qi::graph >>
                                        -(" ponder " >> +qi::graph) >> qi::eoi,
                                    best, ponder);
               };
  
               bool ok = send(_game, yield) //
                   && command("go", bestmove, yield);
  
               if (!ok)
                   throw std::runtime_error("Engine communication failed");
           });
           run_io();
           return best;
       }
  
     private:
       void init() {
           boost::asio::spawn([this](yield_context yield) {
               bool ok = true //
                   &&
                   expect([](std::string_view banner) { return true; }, yield) //
                   && command("uci", "uciok", yield)                           //
                   && send("ucinewgame", yield)
                   && command("isready", "readyok", yield);
  
               if (!ok)
                   throw std::runtime_error("Cannot initialize UCI");
           });
           run_io();
       }
  
       bool command(std::string_view command, auto response, yield_context yield) {
           return send(command, yield) && expect(response, yield);
       }
  
       bool send(std::string_view command, yield_context yield) {
           debug_out << "Send: " << std::quoted(command) << std::endl;
           using boost::asio::buffer;
           return async_write(_sink, std::vector{buffer(command), buffer("\n", 1)},
                              yield);
       }
  
       bool send(MoveList const& moves, yield_context yield) {
           debug_out << "Send position (" << moves.size() << " moves)" << std::endl;
  
           using boost::asio::buffer;
           std::vector bufs{buffer("position startpos"sv)};
  
           if (!moves.empty()) {
               bufs.push_back(buffer(" moves"sv));
               for (auto const& mv : moves) {
                   bufs.push_back(buffer(" ", 1));
                   bufs.push_back(buffer(mv));
               }
           }
           bufs.push_back(buffer("\n", 1));
           return async_write(_sink, bufs, yield);
       }
  
       bool expect(std::function<bool(std::string_view)> predicate, yield_context yield)
       {
           auto buf = boost::asio::dynamic_buffer(_input);
           while (auto n = async_read_until(_source, buf, "\n", yield)) {
               std::string_view line(_input.data(), n > 0 ? n - 1 : n);
               debug_out << "Echo: " << std::quoted(line) << std::endl;
  
               bool matched = predicate(line);
               buf.consume(n);
  
               if (matched) {
                   debug_out << "Ack" << std::endl;
                   return true;
               }
           }
           return false;
       }
  
       bool expect(std::string_view message, yield_context yield)
       {
           return expect([=](std::string_view line) { return line == message; },
                         yield);
       }
  
       void run_io() {
           _io.run();
           _io.reset();
       }
  
       boost::asio::io_context _io{1};
       bp::async_pipe          _sink{_io}, _source{_io};
       bp::child _engine{"stockfish", bp::std_in<_sink, bp::std_out> _source, _io};
  
       MoveList& _game;
       std::string _input; // read-ahead buffer
   };
  
   int main() {
       MoveList game;
       Engine   white(game), black(game);
  
       for (int number = 1;; ++number) {
           game.push_back(white.make_move());
           std::cout << number << ". " << game.back();
  
           game.push_back(black.make_move());
           std::cout << ", " << game.back() << std::endl;
  
           if ("(none)" == game.back())
               break;
       }
   }