M4A files and M4P music files both support the same embedded album artwork facility. While the equivalent feature for MP3 music files uses 'ID3 Tags' for M4A i.e. AAC music files the term used is MPEG4 'ATOMS'.
Windows Media Player supports itself reading not only standard labelling information like Album name, Artist name, Track name, etc. from AAC files, but also any embedded album artwork. It should be noted however that as standard M4A files purchased from the iTunes Store do not contain embedded album artwork, instead when you purchase or download a music file from the iTunes Store it starts off in life by also downloading separately the matching album artwork and storing this (separately) in another set of sub-folders in your iTunes library. It is however very easy to get iTunes to copy that artwork and 'embed' it in the track.
If you are importing your own CDs for example in to iTunes then again initially the resulting M4A files will not contain any embedded album artwork, it is optionally possible to configure iTunes to automatically search for and if a match is found download artwork from the iTunes Store, once again this is initially stored as a separate file and again can be converted to be embedded instead.
Historically Windows Media Player has either itself looked for matching artwork and downloaded it although at a far lower quality than the quality available via iTunes and saved it as file(s) in the same folder as the music tracks, or it has also accepted a folder.jpg file again in the same folder as the music tracks. (The tracks are normally organised with sub-folders per album.) As mentioned WMP now also supports reading embedded artwork from MP3 and M4A files.
What you are describing however relates to Windows Explorer which is the Windows equivalent of the Mac Finder. I have not personally devoted any attention to how this works or can be enhanced but I would certainly expect it to support at least showing the folder.jpg as a thumbnail in the enclosing folder although not per track. I do however seem to vaguely recall briefly using AudioShell to add some capabilities along these lines for Windows Explorer.
See http://www.softpointer.com/AudioShell.htm
