why a calculated result (0.35) is rounded 0.3 instead of 0.4?

Here is the story ...

Cell B4 contains 34.2 (manual input)

Cell C4 contains 34.9 (manual input)

Cell F4 contains the following formula: =0.5*(C4-B4)

The displayed result is, of course, 0.35

When I decrease the number of decimal digits (button in the upper bar), the number displayed is 0.3 instead of 0.4.

I get this error in an unpredictable way; 0, 1, 2 times in the same spreadsheet.

May anyone help me?

Thank you very much.

Paolo

Mac OS X (10.7.2)

Posted on Oct 17, 2012 9:16 AM

Reply
3 replies

Oct 17, 2012 9:40 AM in response to paolo25858

Paolo,


It's quite common for a result on a digital device to be off by 1 in the least significant calculated digit. Apple apparently didn't choose to deal with this to satisfy your need to conform to the rounding conventions. For this reason, the 15th significant digit is unreliable and may lead to rounding errors.


If in cell F4 you write =ROUND(0.5*(C4-B4), 14) you will have thrown out the 15th place and I believe your rounding will behave as you wish. I'm not proposing that you make a practice of rounding all your results, unless the OS X behavior really bothers you. If you are making invoice documents where you calculate fractional amounts in the process of adding taxes or subtracting discounts, it's a very good practice to use ROUND so that the numbers behind the document match exactly what is printed. That way you won't add 0.02 and 0.02 and get 0.05. Strange how customers hate to come out on the short end of a discrepancy of one penny.


Jerry

Oct 17, 2012 10:31 AM in response to paolo25858

The problem, as Jerry said, is the way computers do math. Decimal numbers (i.e., non-integers) do not always translate to exact base 2 numbers. Very small rounding errors occur. There is an IEEE standard for floating point math that says how this is supposed to work. Apple apparently follows the standard and this sometimes leads to results like the one you are seeing. Other applications (such as Excel) do not follow the standard strictly; they change it up a little so the answers fit what you would expect. The advantage of following the standard is that you will get the same result doing the same operation in any app that follows the standard. The disadvantage is you might not get the answer you expected or that you would have gotten if you had done the math in base 10 on a piece of paper.


If you are curious, here is a Wikipedia article on the floating point standard, IEEE 754


http://en.wikipedia.org/wiki/IEEE_floating_point


And here is an explanation on Excel's way of dealing with floating point. Also a good read to better understand the problem.


http://support.microsoft.com/kb/78113

Oct 17, 2012 10:44 AM in response to Badunit

Additional:


I tried your example in Excel. Same thing happened there. I was surprised. Apparently their "optimization" doesn't cover all instances of rounding errors. Inconsistency is a killer, thus the need for standards.


The example below is the one we typically point to to demonstrate the problem. It gives a different result in Excel vs Numbers.


B1=0.1

B2=0.2

B3=0.3

B4=B1-B3+B2

Result is 0 in Excel

Result is 2.8E-17 in Numbers

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

why a calculated result (0.35) is rounded 0.3 instead of 0.4?

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.