Annoyingly iTunes/Music on your computer and Music on iOS don't necessarily handle the same metadata completely consistently. Without knowing more I suspect that you do have some compilations in your library, but don't have an option selected that would highlight them, although there may be other explanations. Select the Songs view of the library, then make sure that View > Column Browser > Group Compilations is ticked. The Compilations section of the library may now show the same set of albums as show up on your iPod.
The compilations flag is used somewhat inconsistently both to mark a collection of tracks by different artists all on the same album (e.g. "That's What I Call Music #451") and an anthology of a single artist's work from across their career (e.g. "Greatest Hits"). The mechanism in iTunes is functionally more useful when only applied to the first type of album. In either case populating Album Artist can help link the tracks of album together for best results.
See http://samsoft.org.uk/iTunes/grouping.asp for further background.
tt2