The link posted by Scott is an excellent summary.
The basic advantages are as follows:
1) 24 bit allows a much better resolution at lower volumes. Analog to digital converters work by changing an analog voltage (which is related to amplitude of the incoming signal) into a digital signal. At 16 bit, you have 65536 (2^16) distinct digital representations of those voltages. However, most of these are clustered around the top of the signal range. For instance, if you use up 15 of those 16 bits, you have 32768 steps remaining for the final bit. However, you have the same number of steps to represent the previous 15 bits of information. That is, you have the most accurate representation of the analog signal when using most of the available bits (i.e. louder = better).
In 24 bit, the same logic applies, but you have 16777216 (2^24) steps, and 8388608 steps for the first 23 bits. In fact, to get the same resolution as 15 bits full in a 16 bit system, you can have a signal that is 8 fold lower volume (23-15=8). So while it is still true that you get the best resolution as you 'use up the bits', you dont have to worry about this so much in a 24 bit system. This is why if you can, YOU SHOULD ALWAYS USE 24 BIT. Read that again.
2) 44.1kHz vs. 96kHz is MUCH more confusing. There are two reasons to use 96kHz, and one main one not to. The basis of chosing a sampling rate is that it MUST be at least twice the maximum frequency in your source. ALL digital converters use a brickwall filter to eliminate signals which are higher than 1/2 the sampling rate. Since human hearing only extends to 20kHz, the argument is that 44.1kHz is sufficient to cover all possible frequencies and the filter removes higher frequencies. However, many instruments produce overtones greater than 20kHz, which affect the perception of sound, so it can be argued that increasing the sampling rate allows those overtones to be 'heard' (or felt, as the case may be), and thus improves subjective sound quality. Some people claim they can hear the difference.
The second (and IMHO, best) reason to use 96kHz is that even if we can only hear up to 20kHz, the use of 96kHz (or even 88.2 kHz) allows a MUCH softer filtering of ultra-sonic frequencies. This filter can negatively impact audible sound quality (just listen to early CD players). If the cutoff frequency is at 48kHz (for 96kHz sampling), that is WELL beyond out hearing range (more than one full octave), and thus the filter can be much gentler, and thus less detrimental to sound quality.
HOWEVER, there is one HUGE drawback to 96kHz. It uses resources like crazy. You will get significant reductions in processing power using 96kHz, since they are much more complicated files for the CPU to work with.
IMHO, until you have a computer than can really handle the extra requirements, go with 24/44.1. MANY top notch records have been made at 24/44.1, and you'll get the benefit of 24 bit without the drawback of 96kHz.
Mike