I think it all depends on the purpose of the video player. I spend a lot of time on different discussion forums and debate on similar topics, I rarely see anyone complaining about VLC because of its performance quality. However, I agree about the interface, I would really like to see an updated modern UI, although I don't think such a miracle will happen in the near future.
As for alternatives, you need to consider a lot of points, what number of formats the player supports, its functionality, and more individual is the UI, in fact, there is a lot to choose from. Elmedia Player fits my needs well. Any format is played as native, nice and simple interface, subtitle support, AIrplay, generally a lot of useful features that I use.
If I were to approach the question fundamentally, I would choose several options and go read reviews or compare functionality. It is important to learn what real users think to save yourself time testing 3-4 different players on your own. You might also consider Infuse or Omniplayer, which are also well integrated players for MacOS.
One thing I've noticed about different players is how they handle fast-forwarding and frame-stepping. Some are super smooth, while others lag or take a second to catch up. If you do a lot of skipping through videos, this can be a big deal - some just feel more responsive than others. It may not seem like a big deal at first, but it can become annoying over time. Good luck with your choice!