Q: How do I get the changes made in sounds to stay Permanently?

Hello there, Mer-aps.

It sounds like you are on the right track and following the information found in this Knowledge Base article:

iOS: Understanding sounds, ringtones, and alerts

http://support.apple.com/kb/HT5800

There is another article that provides additional understanding of Notification Center which holds the other piece of the puzzle:

iOS: Understanding notifications

http://support.apple.com/kb/ht3576

Verify the sound is also set to what you want here (in your case Mail is the app name):

Settings > Notification Center > <app name>.

Thanks for reaching out to Apple Support Communities.

Cheers,

Pedro.