Reinforcement Learning Agent in FMU

Question

I want to train a reinforcement learning agent on a model which i build in OpenModelica. By using pyFMI, it is no problem to import the FMU, simulate it, and get some results.

My problem is that i don´t have a possibility to "pause" a simulation after each step, getting the states, feeding my RL-agent with it and returning his proposed action as an input.

ModelicaGym seems to be a way to solve this problem by starting a simulation, stopping, getting the results, defining the next action and starting the simulation again with the last end-time as starting time.

Reading a paper from Lund University (https://portal.research.lu.se/portal/files/7201641/pyfmi_tech.pdf) made me think about an other idea:

Creating a FMU with the Learner, and connecting the two FMUs via PyFMI.Master.

Something along these lines:

from  pyfmi  import  load_fmu
from  pyfmi.master  import  Master

controller   = load_fmu("controller.fmu")
Circuit = load_fmu("circuit.fmu")

connections = [( Circuit ,"currentSensor1.i",controller ,"feedback1.u2"),
               (controller ,"PID.y",Circuit ,"signalVoltage1.v")]

models = [Circuit , controller]
master_simulator = Master(models , connections)
res = master_simulator.simulate(final_time =1)

Controlling the circuit with an other FMU with a PID controller inside works, but is it possible to create a FMU with a Reinforcement Learning Agent, including all other requiered Libraries, packages (Keras, Tensorflow?)

According to my point of view, such an implementation could have a pretty good performance, especially for models and learners with a higher complexity, this could be an interesting approach.

Or am I just chasing some dreams, because implementing a Reinforcement Learning algorithm in a FMU is not possible or causing other troubles?

Actually, i was a little surprised of not finding other people trying to implement this.

Best regards

Henrik

Answer 1

This answer might be plentifully late, but nevertheless I found your question during my research for the exact same problem. Your question is - to my understanding - taken up in the paper

Integration and Evaluation of Deep Reinforcement Learning Controller in a Building CoSimulation Environment [PDF found here]

However in our context, the co-simulation environment that is used for the study of Cyber-Physical Systems is a master of co-simulation, and need to be able to call AI based Python libraries or tools from within the co-simulation. In the co-simulation, the controller requires an interaction with the controlled component at each iteration. Therefore, it is necessary to generate the AI based control components in the form of an FMU as is the case for the other components.

They used a tool called DACCOSIM NG but later introduced their own methodology to make this approach more streamline.

Hope you long since found your own solution.

Answer 2

maybe you can update your question so it is more clear how the learning agent is implemented, but I understand that it can be used from Python?

The example fmu_with_input_function.py from the PyFMI documentation illustrates how to use a function as input an FMU. I suppose you can retrieve information from the FMU in this function like so (untested pseudo code):

from pyfmi import load_fmu

define input_object(model):
  response = model.get('response_variable_name')
  return ('input_var_name', learner(response))

model = load_fmu('model.fmu')

res = model.simulate(final_time=30, input=input_object(model))

You have to set up your model FMU so that the variables your learner should change (input_var_name), are input variables or tunable parameters. If you use parameters without variability="tunable", you cannot change them in the course of the simulation.

I would first try with input variables, because tunable parameters are a bit more complicated to treat and might not be implemented correctly in some tools.