The issue you're running into is that apple has adopted ISO standards for recording Date and Time values. Any time your see displayed in Numbers is a Date and Time value, with only the Time part displayed. Any Date you see displayed in Numbers is a Date and Time value, with only the Date part displayed. No matter which part is displayed, a reference to the cell will always return the full Date and Time value.
When you enter a Date, Numbers automatically sets the Time part of the value to 00:00:00 (midnight, at the beginning of that date).
When you enter a Time (of day), Numbers automatically creates a full Date and Time value by setting the date part to 'today's date'. (This is the setting that affects your calculations.)
The base unit for Date and Time value is the day, so there are ways to correct for this apparent error.
Here is your basic formula for calculating the Duration between two Date and Time values where B2 contains the Start time, C2 contains the Finish time, and both are on the same calendar day:
=C2-B2
The result will be a Duration value
To avoid having to correct for a start time before midnight and a finish time after midnight, the straightforward solution is to enter those times during the calendar day on which each occurs. That's straightforward, but not always convenient.
If both times are entered on the same calendar day, you can apply a correction factor to the formula above:
=IF(C2<B2,C2+1-B2,C2-B2)
The result will be a positive Duration value.
Taking it one step further, you can use TIMEVALUE to strip the Date part from a Date and Time value, leaving a decimal fraction representing the portion of the day that has passed.
=IF(C2<B2,1+(TIMEVALUE(C2)-TIMEVALUE(B2),TIMEVALUE(C2)-TIMEVALUE(B2))*24
The result is a number representing the number of hours between the two times.
Here's a table showing five examples, and the results of the formulas above:

Regarding: "I also know to enter the times as "23:00" and "03:25", but do I have to do that??"
Yes. Numbers will not recognize a (date and) time entry without the colon.
Regards,
Barry