IBM Support

Decimal precision in Cognos 10: why is precision sometimes limited to 15 digits?

Technote (troubleshooting)


Problem(Abstract)

For databases that support large-precision decimal values, Cognos 10 only supports up-to the
precision of double-precision floating point values, or the equivalent of 15 significant digits.
Digits beyond this will be rounded in some but not necessarily all cases, depending upon
the database used, and the structure of the report.

Cause

Databases supported by Cognos 10 often contain support for large-precision decimal values.

The level of precision varies among different vendors, but Oracle and SQL Server, for example,
support a number format that allows 38 decimal digits to be stored without loss of precision.

Cognos 10 can access and correctly display these values, but when values are aggregated or
calculations are performed on these values, the precision may be reduced to double-precision
floating point precision. For example, list reports using decimal values with a precision
larger than 16 digits from an Oracle database will be aggregated using double-precision
floating point arithimetic rather than the full precision stored within Oracle.

This is currently a restriction due to the way Cognos 10 accesses Oracle data. In the case of
SQL Server, data can be retrieved at full precision and results calculated at the full
precision because the API used by Cognos 10 to access SQL Server databases supports this.

The double-precision floating point format used internally is the IEEE-754 format, a standard
available on all platforms supported by Cognos 10. Values are stored as 64-bit data items, 52
bits of which are used to store the significant digits. Since 2^52 is roughly 4.5 x 10^15,
integers between 0 and this value can be stored safely without loss of precision. Values beyond
this will be rounded to a maximum accurate precision of 15 digits.

This is the same way many other reporting products and desktop tools like Microsoft Excel store
these values.

To consider this in context, 15 significant digits is usually enough to accurately represent most
values used in reporting. The total government debt of the US government as of Oct. 2005 was
only $8.0 x 10^12 so the value can be represented accurately down to the $0.01 level without any
loss of precision.


Cross reference information
Segment Product Component Platform Version Edition
Business Analytics Cognos Business Intelligence

Historical Number

1015504

Document information

More support for: Cognos Business Intelligence
Analysis Studio

Software version: 10.2, 10.2.1, 10.2.2

Operating system(s): AIX, HP-UX, Linux, Solaris, Windows

Reference #: 1335843

Modified date: 14 May 2011