MDP - techniques generating transition probability

Question

I am working a MDP car supply and demand problem as follows and was thinking whether there is any techniques to generate the transition probability matrix automatically rather than manually.

Assume the demand is as follow:
time, station1, station2
1000, 3, 1
1030, 3, 1
1100, 2, 3

Assume for car from station1, there is a 60% chance that the car will be drop off at station1 and 40% chance to drop off at station2. Assume for car from station2, there is a 80% chance that the car will be drop off at station1 and 20% chance to drop off at station2.

I have calculated manually that the following.

At time step 1,

P(car at station1 = 2,car at station2 = 8) = 0.0432
P(car at station1 = 3,car at station2 = 7) = 0.2016
P(car at station1 = 4,car at station2 = 6) = 0.1344
P(car at station1 = 5,car at station2 = 5) = 0.0896
P(car at station1 = 6,car at station2 = 4) = 0.0512

Hence, will like to check whether anyone could provide insights to calculate the probability at time step 2 automatically, rather than to compute by hand.

For your advises pls.

Answer 1

I'm not sure I understand your question.

If in a stationary Markov Process, the distribution of the state variable x_t (here, the station at which the car) at a given time t is a function only of the transition matrix P and the state at the time t-1.

You can write x_t = x_{t-1} * P for any t, which means that x_t = x_0 * P^t.

Knowing x_0 (the distribution of car at the start, e.g. if the cars are evenly distributed between the two stations x_0 = [0.5 0.5]) and using P = [ 0.6 0.4 ; 0.8 0.2 ], you then get the distribution of cars at any time t > 0 as x_t = x_0 * P^t.