Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Incorrect result in numbers

ROUNDDOWN(1,0231;4) = 1,023 (incorrect)

ROUNDDOWN(1,0232;4) = 1,0232 (correct)

MacBook, OS X Mavericks (10.9.2)

Posted on Mar 26, 2014 5:09 AM

Reply
5 replies

Mar 26, 2014 5:49 AM in response to Ncsft

Maybe...


Overall I think this may be a problem with how numbers are stored internally.


Here's why... The value (column A) ranges from values that start before the number you noticed as "weird" (1.0231) and goes after. Column C uses the formula rounddown(A, 4), and column B computes the absolute value of the error between the two.


Value

Error

Rounded Down

(4 places)

1.0195

0.0001

1.0194

1.0196

0.0001

1.0195

1.0197

0.0001

1.0196

1.0198

0.0001

1.0197

1.0199

0.0001

1.0198

1.0200

0.0001

1.0199

1.0201

0.0001

1.0200

1.0202

0.0001

1.0201

1.0203

0.0001

1.0202

1.0204

0.0001

1.0203

1.0205

0.0001

1.0204

1.0206

0.0001

1.0205

1.0207

0.0001

1.0206

1.0208

0.0001

1.0207

1.0209

0.0001

1.0208

1.0210

0.0001

1.0209

1.0211

0.0001

1.0210

1.0212

0.0001

1.0211

1.0213

0.0001

1.0212

1.0214

0.0001

1.0213

1.0215

0.0001

1.0214

1.0216

0.0001

1.0215

1.0217

0.0001

1.0216

1.0218

0.0001

1.0217

1.0219

0.0001

1.0218

1.0220

0.0001

1.0219

1.0221

0.0001

1.0220

1.0222

0.0001

1.0221

1.0223

0.0001

1.0222

1.0224

0.0001

1.0223

1.0225

0.0001

1.0224

1.0226

0.0001

1.0225

1.0227

0.0001

1.0226

1.0228

0.0001

1.0227

1.0229

0.0001

1.0228

1.0230

0.0001

1.0229

1.0231

0.0001

1.0230

1.0232

0.0000

1.0232

1.0233

0.0000

1.0233

1.0234

0.0000

1.0234

1.0235

0.0000

1.0235

1.0236

0.0000

1.0236

1.0237

0.0000

1.0237

1.0238

0.0000

1.0238

1.0239

0.0000

1.0239

1.0240

0.0000

1.0240

1.0241

0.0000

1.0241

1.0242

0.0000

1.0242

1.0243

0.0000

1.0243

1.0244

0.0000

1.0244

1.0245

0.0000

1.0245

1.0246

0.0000

1.0246

1.0247

0.0000

1.0247

1.0248

0.0000

1.0248

1.0249

0.0000

1.0249

1.0250

0.0000

1.0250

1.0251

0.0000

1.0251

1.0252

0.0000

1.0252

1.0253

0.0000

1.0253

1.0254

0.0000

1.0254

1.0255

0.0000

1.0255

1.0256

0.0000

1.0256

1.0257

0.0000

1.0257

1.0258

0.0000

1.0258

1.0259

0.0000

1.0259

1.0260

0.0000

1.0260

1.0261

0.0000

1.0261

1.0262

0.0000

1.0262

1.0263

0.0000

1.0263

1.0264

0.0000

1.0264


User uploaded file


Rounddown should tend towards zero. That is, the result should always be the same, or closer, to zero. I think, like you, that the vlues you are using with the number of digits should produce the same value. There maybe a reason they do not.


the number 1024 is a power of two, so (1024 - 1)/1000 may be one of those numbers that is imperfect when stored in a limited precision digital number.


You can post feedback to Apple using the menu item "Number > Provide Numbers Feedback"

Incorrect result in numbers

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple ID.