It might be both.
If his apps, music and other purchased media are automatically downloading to your phone, it may be because you are using the same Apple ID in Settings>iTuens & App Store and you have Apps, Music, etc. turned on under Automatic Downloads. To stop this, turn off these items under Automatic Downloads.
If you are getting each other's contacts and other iCloud data on your devices it's because you are sharing the same iCloud account in Settings>iCloud. To fix this, one of you needs to migrate to a separate iCloud account with a separate ID (you can do this and continue to share the same ID for iTunes).
To migrate to a new account, start by saving any photo stream photos you wish to keep to your camera roll (unless already there) by opening your my photo stream album, tapping Select, tap the photos, tap the share icon (box with upward pointing arrow), then tap Save to Camera Roll. If you are syncing notes with iCloud, you'll need to open each of your notes and email them to yourself so you can later copy and paste the text into new notes created in your new account. Then go to Settings>iCloud, tap Delete Account (which only deletes it from this device, not from iCloud; the device keeping the account will not be effected by this), choose Keep on My iDevice and provide the password to turn off Find My iPhone. Then sign back in with a different Apple ID to create your new account and choose Merge to upload your data. The new account will be assigned a new @icloud email address when you turn on Mail.
Once you are on separate accounts you can each go to icloud.com and delete the other person's data from your account.
If you are sharing the same ID for iMessage and FaceTime you will also end up getting each other's text messages and FaceTime calls. If you are, the person that migrated to the new iCloud account should go to Settings>Messages>Send & Receive, tap the ID, sign out, then sign back in with their new iCloud ID. Then do the same thing in Settings>FaceTime.
