There is nothing that should cause such a noise if the system is otherwise inactive, unless there is a CD or DVD in the optical drive which the system briefly spins up - and even that would depend on what is on the disk.
Since it can be rather difficult to tell whether this kind of noise is the optical drive or fan, since they can make similar sounds, I would first see if the system makes this noise at any point after being booted when there has not been a disk in the optical drive at all since the startup. If no disk has been inserted, the optical drive should not have any reason to spin up. If the noise still occurs, then download a copy of MenuMeters (
http://www.ragingmenace.com/software/menumeters/) set it to display all counters including the one for paging, and then monitor the system. As and when the sound recurs, check if there is any sign of high CPU or prolonged drive activity at the same time. If so, then the sound is likely the fan being spun up to deal with load on the system, and it's a question then of finding what it is that is causing the activity, by using Activity Monitor (in Applications>Utilities). If there is no high CPU or disk usage, then the sound is most likely a fault condition, and if the mini is the unit outlined in your profile (ie, a core 2 duo 2.0HGz model) then if it were mine, I would contact Apple to discuss the problem, or take the system to an Apple store or service provider to have it looked at.