Previously I managed to get working vlcj in jme3 in a quite decent way. Now I switched the code to a older machine, single core, and the thing is terrible! The video won't even play properly, and after 15-20 sec the video just crashes, and restarts.
When I open the video with vlc player, or vlcj with the regular java jframe window it works just fine.
Is there anything that could be done to optimize the thing so that it can work on older machines?
I don't think you are going to get much luck here. Rendering a 3d game is intensive, rendering video is also intensive.
If the hardware just doesn't have the power to cope then all you can do is try and reduce the load by (for example) reducing the screen resolution, lowering the video quality, etc. For example if you have a separate video file at 1/2 the resolution in each direction and 1/2 the framerate and it will use 1/8th of the resources.