Download and install Windows Media Preview from here.

Choose the video files for which you need thumbnails. Next time you open the folders, you will see previews