How to do model agnostics on LSTM neural network in Python?

Question

I used an LSTM neural network (Keras package) on my data set and want to use model agnostic methods to find the effects of the variables on my prediction. The data set is an array with three dimensions. I have multiple ID's that each have their own sequence, with variable timesteps for each ID. The shape of my data set looks like this: (18610, 151, 18) (ID's, time-steps, variables) and the variable output for one ID on in one time step looks like this (with prediction class 1):

[2.00000000e+00 0.00000000e+00 2.10544316e-03 1.56082309e-03
 0.00000000e+00 1.01647775e-01 0.00000000e+00 4.28571559e-01
 6.95422174e-01 8.52459016e-01 1.20027614e-01 1.00000000e+00
 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
 0.00000000e+00 1.00000000e+00] 1

All model agnostics methods (as far as I could find) do not work on these array data sets with three dimensions. Is there a way to still find the effects of the variables (using Lime, partial dependence plot or Shapley?) so I can interpret the model? I would prefer using a partial dependence plot to look at the global interpretation.

Does anyone know how to do this (with example code)? Thank you so much in advance!

Answer 1

I am not sure for what purpose you need the model agnostic methods, but I use one for feature ranking. The shape of your data (18610, 151, 18) is of 3D. You can use SHAP method which is model-agnostic and it is ideal for your data.