OSX Calculator error ?

Question

Level 1

4 points

OSX Calculator error ?

Using 10.4.10 with latest Intel audio fix, I tried a very simple multiplication with Calculator (ver 4.0.5).

101661.15 x 2 =

Instead of displaying 203322.3 calculator is displaying 203322.29999999999

I've tried subsets of the number (ex: 661.15 x 2) and all come back with a result with .3 at the end. It seems only the whole sequence returns the factional result.

Anyone else seen this before? I did a search and nothing came up.

Mac Mini Intel Mac OS X (10.4.10)

Posted on Jul 4, 2007 1:05 PM

Reply

Answer 1

Jim Beggans Author

Level 1

4 points

Jul 4, 2007 1:21 PM in response to Jim Beggans

It looks like setting precision below 10 will correct this issue. Still that is weird.

Reply

Answer 2

Jul 4, 2007 1:29 PM in response to Jim Beggans

HI Jim and Welcome to Apple Discussions

Try this... locate this file: com.apple.calculator.plist
It's here: Home/Library/Preferences. Open your Preferences folder, locate that file, drag it to the trash and reboot. See if Calculator works properly... if not, post back.

Carolyn 🙂

Reply

Answer 3

Jul 4, 2007 1:36 PM in response to Jim Beggans

The calculator seems to be working with 64 bit binary numbers internally. 101661.15 does not convert exactly to a binary number, so results won't be exact. Normally the rounding hides this. You probably have Precision (View menu) set to 11. If you reduce it you won't see the error. To see what is actually happening, set Precision to 16. Then enter 101661.15 and multiply by 2. You will see the result with more digits, and can see that if rounded to 11 decimal places it will give 203322.29999999999, but rounded to 10, or less, will give 203322.3, which is what you want.

Any calculator that uses binary internally can have this problem with decimal numbers, but some hide it better than others, such as by not displaying results with more precision than the binary conversion of the inputted decimal values.

Reply