TextEdit's behaviour is very strange - if you change the default new document to "Plain Text" then create a new document you get the option to chose the character set to use when saving, but no option to pick RTF, DOC etc.
If you change the default new document to "Rich Text" then create a new document you get the option to pick RTF, DOC etc but no option to save as plain text.
If you change the preferences it doesn't impact the behaviour of the open & unsaved window. You have to create a new document after chaning the preferences.
Easier to give up on TextEdit and use somethng like TextWrangler which just works like you'd expect it