If it is doing this fully automatically then it is something you will have to change in your browser's preferences application settings. You can set this to do certain things with certain file types and exactly how depends upon the browser you use. Rght now you probably have it set to automatically open all files of type mp3 (or whatever) with iTunes after downloading.
If it is only doing this once the file is downloaded but you click on it to open it then you can change the preferred player to a different one.
Mac OS X: Double-Clicking a File Opens the Wrong Application - http://support.apple.com/kb/TS2291
-= Changing the application used to open all files of a certain kind =-
1) In a Finder window highlight a file of the kind you want to change the application to open that kind of file.
2) While that file is highlighted, select File > Get Info or press command (apple or propeller icon) + i to get a file information window.
3) In the lower part of the info window there is an "open with" menu with a list of applications.
4) If your application is already in the box then it is the default application for opening that kind of file and you don't need to do anything more. Close the get info window.
5) If the application showing in the menu is not the one with which you wish to open the file then select a new application. If your application does not appear there then select the "other..." and track down the application (usually in the Applications folder at the main level of the computer).
6) If you wish to change all files of this type to open with this application in future, make sure the "change all" button is selected.
7) Close the get info window.