Seems like the VLC update from Raspbian broke something. Your problem seems similar to a recently reported issue here. I'm going to ask the same things to track things down
add the info requested in https://retropie.org.uk/forum/topic/3/read-this-first run the ffmpeg command to identify which video codec/format is the video file that doesn't work.Can you upload on of the video files with the problem somewhere (Dropbox/Google Drive/etc.) to take a look at it ?