change the tempos of the project from 120 to 60

When I change the tempo of the project from 120 to 60, the audio file always follows the measure, 
which makes the audio files have a lot of silence. 
Does anyone know how to make the audio not follow the measure and only follow the time when changing the tempos?
Thanks :)