 |
Software > Lotus >
|
|
|
|
Floating point arithmetic and rounding errors in Notes
|
| | | | Question | | How is floating point arithmetic implemented in Lotus Notes® and is it normal to see inaccuracies in calculations? | | | | | Answer | Floating point arithmetic is implemented in Notes according to the Institute of Electrical and Electronics Engineers, Inc. (IEEE) Standard for Binary Floating-Point Arithmetic (IEEE 754). This is the standard used by most PC-based software that implements floating-point math and by many co-processors.
The IEEE standard requires that numbers be stored in binary format. However, not all values can be represented exactly in binary within a finite space and inaccuracies can occur when these binary representations are converted back to decimal.
Example:
The decimal number 0.2 when converted to binary is gives 0.001100110011001100110011001100110011(...repeating).
Converting this back to decimal, taking into account 32 bits for example, will give: 0.19999999995343387.
Workaround:
The @Round function can be used to round any decimal values to a significant digit. For example:
@Integer(6000*2.3) =13799 <- inaccurate
@Integer(6000*@Round(2.3, .0001)) =13800 <- accurate
The IEEE standard offers a number of advantages such as limiting the memory required to store numbers and enabling the rapid processing of data. However, rounding errors as in the example above are part of the "cost" of this implementation.
In the case of such rounding errors, Notes is considered to be working as designed. Most PC-based software implements the same IEEE standard and is susceptible to the same errors.
For additional information on IEEE 754, refer to the following link:
http://grouper.ieee.org/groups/754/ | | | | | | | | |
|
|
| IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml. |
|
|
|
|
| Please take a moment to complete this form to help us better serve you. |
|
 |
|
|
|
|
|
|
| Product categories: |
|
| | Software | |
| | Messaging Applications | |
| | Advanced Messaging | |
| | Lotus Notes | |
| | Lotus Notes | |
|
| Operating system(s): |
| |
Linux, Windows
|
|
| Software version: |
| |
5.0, 6.0, 6.5, 7.0, 8.0
|
|
| Reference #: |
| |
1238705
|
|
| IBM Group: |
| | Software Group |
|
| Modified date: |
| | 2009-05-29 |
|
|
