Using BOP-DMD (Bagging, Optimized Dynamic-Mode Decomposition) to predict next frame in Moving MNIST dataset

Question

I've been trying to use BOP-DMD to predict next frames in the Moving MNIST dataset.

Copying what's already online I can upload and visualize the frames (moving MNIST) and I'm using them as the input for my X matrix. As for the BOP-DMD I've been using the implementation contained in PyDMD.

What I am doing is this:

1. I load 1 64x64 frame as 1 column (1 column of 4096=64*64 rows) (4096,1)

2. I put 8 frames (frame 0 to frame 7) this way constructing my X matrix. (4096,8)

3. I use BOP-DMD so I need a time vector. I use t=[0, 1, 2, 3, 4, 5, 6, 7]

4. I solve (fit) using

                  from pydmd import BOPDMD
                  optdmd = BOPDMD(svd_rank=15)
                  t = np.linspace(0, 7, 8)
                  optdmd.fit(X,t)
   

5. Then I forecast using: Y = optdmd.forecast(t) This should (and it does) replicates the frames used in the fitting, as I'm using the same t. The images are somewhat worse, but I attribute that to the fact that this is an approximation of all those dimensions trying to be fitted into a finite space (svd_rank=15).

My problem is when I try to Forecast for t={1:8} (advancing 1) or advancing any number of frames into the future. The result does not match at all! I have up to 20 frames but I intentionally used the first 7 and want to check number 8

My question is has anyone done something like this? I know I'm doing something wrong but I can't seem to find the mistake.

Moving MNIST: https://www.cs.toronto.edu/%7Enitish/unsupervised_video/ BOP-DMD: https://www.youtube.com/watch?v=-VENSFxJstU

Any help would be appreciated.