OK found a solution!
It is really lame!
NEVER add a reminder in iCal dragging them into the Week or Day view!!!
Change to Month view and then drag the reminder on the desired day!
You can now change to Week or Day view and move the reminder to the selected hours.
Please help share this solution and click like to get Apple to finally fix this issue, which is clearly on their side.
You're talking about how to avoid the problem from happening in the future, or for those who has not encountered such problem.
What about the fix to those, like me, who already suffers from such annoying problem for months? (I have tens of calendars, thousands of calendar (diary and future) events, hundreds of reminders already...)