This delay was driving me nuts until I figured out to turn off the double-tap-to-drag option. It's just really poor UI design to allow a delay like that on every tap. There's no need for it to wait for the second tap -- just engage dragging if a second tap arrives in time on a draggable area. If the tap area cause any action due to a single tap (e.g. a button), then that isn't a draggable area without clicking the trackpad button. This is simple stuff, and a bad oversight by their interface design folks.
Also, the trackpad needs a slider setting for tap sensitivity. It is starting to give me user-rage that half my taps aren't being recognized, although at least with the dragging setting turned off I don't have to wait to find out which were recognized and which weren't.
Third, I agree with the person who said the tactile response of the integrated trackpad button isn't very nice. I am surprised Apple chose a button solution with such poor user feel.
I bought this mac (my first) for the trackpad and multi-touch and gestures more than any other feature, and these issues are really disappointing. Lest it sound like I'm just here to bash Apple, I like the computer otherwise. These happen to be the few issues that are really frustrating me, and which are surprising given Apple's emphasis and outstanding record with this sort of user design.