After using ffmpeg to remove some streams in mkv file, it takes much longer to open the media file in potplayer in Windows

Question

It is well known that ffmpeg command line can be used to remove some streams in mkv or mp4 files, and I have no problem in doing so.

ffmpeg -i input.mp4 -map 0 -map -0:a:3 -map -0:a:6 -c copy output.mp4

The only issue I meet sometimes is, the result media files in mkv or mp4 are much much slower to be opened by player. I cannot figure out the real reason and bypass it.

Any hint or help on solving this issue? Thanks!