Is there anyway to do proper action masking while using Ray and an Open Spiel environment?

Question

Whenever I run an Open Spiel environment with Ray, I always get tons of errors about the moves not being legal. I was wondering if there was anyway to apply action masking with an Open Spiel environment.

I ran the example included with ray: https://github.com/ray-project/ray/blob/master/rllib/examples/self_play_with_open_spiel.py

I kept getting this error:

(RolloutWorker pid=27106) OpenSpiel exception: /project/open_spiel/games/connect_four.cc:94 CellAt(kRows - 1, move) == CellState::kEmpty (RolloutWorker pid=27106) CellAt(kRows - 1, move) = X, CellState::kEmpty = Empty

Answer 1

The ActionMasking example does just that. You can use ActionMaskModel to avoid illegal actions. Be aware that such a model always needs access to the actual illegal actions. RLlib itself has no knowledge of the specifics of Open Spiel environments. So in order to successfully mask them, you'll have to wrap the environment.

• This example should include all you need.
• There is also a user implementation out there.