Apple Music Family Plan on HomePod

My family is debating getting a set of HomePods, and so my question is as follows: If we had 3 HomePods downstairs, and I had my own HomePod in my room, is there a way to make it so all the HomePods except my own don't have access to my Apple Music Library? I want a connected smart speaker family in the house, but don't want others listening to my music. Any help is appreciated! Thanks!