SeldonAPI error for mlserver model deployed on seldon

Question

I am trying diagnose a model invocation error seen below. The error is random (5-10% traffic). Seldon system is not constrained by memory or cpu. Anyone has an idea what this stacktrace implies?

level=error logger=SeldonRestApi thread=<no value> message=<no value> stacktrace=github.com/seldonio/seldon-core/executor/api/rest.(*SeldonRestApi).predictions
    /workspace/api/rest/server.go:330
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2047
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func1
    /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2047
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func1
    /go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2047
github.com/seldonio/seldon-core/executor/api/rest.handleCORSRequests.func1
    /workspace/api/rest/middlewares.go:64
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2047
github.com/gorilla/mux.CORSMethodMiddleware.func1.1
    /go/pkg/mod/github.com/gorilla/[email protected]/middleware.go:51
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2047
github.com/seldonio/seldon-core/executor/api/rest.xssMiddleware.func1
    /workspace/api/rest/middlewares.go:87
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2047
github.com/seldonio/seldon-core/executor/api/rest.(*CloudeventHeaderMiddleware).Middleware.func1
    /workspace/api/rest/middlewares.go:47
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2047
github.com/seldonio/seldon-core/executor/api/rest.puidHeader.func1
    /workspace/api/rest/middlewares.go:79
net/http.HandlerFunc.ServeHTTP
    /usr/local/go/src/net/http/server.go:2047
github.com/gorilla/mux.(*Router).ServeHTTP
    /go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210
net/http.serverHandler.ServeHTTP
    /usr/local/go/src/net/http/server.go:2879
net/http.(*conn).serve
    /usr/local/go/src/net/http/server.go:1930