This is usually a sign that your headphones are not plugged all the way in. Make sure the white part of the headset is flush with the body of the Nano and no silver from the plug can still be seen.
If you are 100% positive that the earbuds are all the way in, double check to make sure the headphone jack is clean and clear of debris and/or damage.
B-rock