Unwanted Apps

I am annoyed that "game center" and "you tube" have been installed on my iPhone4 without my requesting them.

YouTube, and many other apps, were installed at the factory (without your permission).

How can I delete them permanently

They cannot be removed, but can be moved to an out of the way screen so you are not forced to look at them.

and how do I prevent further ones being added?

Never update again.

To remove YouTube from your homescreen, you can also disable it like so:

Settings > General > Restrictions > Youtube.

Game Centre cannot be disabled, but can be tucked away in a folder.