I finally figured out something.
You can try this at home. The viewport tag is not respected for me when ran locally. However, I uploaded the same page to my website and tried it there and when you access the page from the website it DOES work. I got the idea from some article I found about iPad not respecting viewport if it's local or something.
What I can get to work: user-scalable, initial-scale
What doesn't seem to change no matter what I do: width, height