"which other formats are available at iTunes store and how do I download them instead of m4a"
None. All music purchases from the iTunes Store are in AAC (m4a) format, almost all at 256kpbs. As noted, other online retailers such as Amazon provide MP3 files (typically varying between 256-320kbps)
"(I don't like how m4a sound in modern mixes that use distortion, although it's closer to analog than mp3, technically it's inferior)."
Actually, AAC is technically superior to MP3, though that is really only measurable or perceptible at lower bit rates (128kbps or lower). If you're concerned about audio quality you should probably avoid all lossy formats and get your music from CD or other lossless sources ... though not mainstream, there are some online retailers that sell lossless audio, typically in FLAC format, either in CD encoding (16bit, 44.1kHz) or higher bit depths/sampling frequency up to 24/192. Of course, if you really like the limitations and distortion of analog sources, you can always stick to vinyl 😁.