IBM Support

ITCAM Extended Agent for Oracle Database: 6.3.1-TIV-ITM_EXT-FP0002

Download


Abstract

This fix resolves the APARs and defects and provides the enhancements listed
in the "Problems Fixed" section below.

Download Description

(C) Copyright International Business Machines Corporation 2012, 2013.
All rights reserved.

Component: IBM(R) Tivoli(R) Composite Application Manager Extended Agent for Oracle Database,
Version 6.3.1

Component ID: 5724V09OR

Fix: Fix Pack 0002, 6.3.1-TIV-ITM_EXT-FP0002

Date: 30 August, 2013

Table of contents:

1.0 General description
2.0 Problems fixed
3.0 Architecture and prerequisites
4.0 Image directory contents
5.0 Installation instructions
6.0 Additional installation information
7.0 Known problems and workarounds
8.0 Additional product information
9.0 Notices

1.0 General description
=======================
This fix resolves the APARs and defects and provides the enhancements listed
in the "Problems Fixed" section below. This fix also includes the superseded
fixes listed in section 2.4. This fix pack contains only updates for the
Extended Agent for Oracle Database.

Download the installer images from the Passport Advantage web page:
http://www-306.ibm.com/software/howtobuy/passportadvantage/index.html

Physical Part Number: ITCAM Extended Agent for Oracle Database CBAQ2ML
Electronic Part Number: ITCAM Extended Agent for Oracle Database CIL7KML

When this fix pack image is downloaded from Passport Advantage, these rules
apply:
* You can locally install the fix pack on a system that already hosts a
licensed copy of IBM Tivoli Monitoring for Databases.
* You can locally install the fix pack on a system that does not already host
a licensed copy of IBM Tivoli Monitoring for Databases.
* You can populate agents to the depot.
* You can remotely deploy agents to existing or new installations.

2.0 Problems fixed
==================
The following problems are addressed by this fix.

2.1 APARS:
----------

APAR : IV24938
Abstract : WRONG PCT_FREE CALCULATION IN RDB_RECOVERY_FILE_DEST
Additional Information: The Oracle Database Extended agent displays a wrong
value for Free Space(%) in the following workspace. The value
that is displayed is less then the real value.

Archive Node -> Flash Recovery workspace ->
Flash Recovery Space Usage view

APAR : IV26915
Abstract : ORA-12504: TNS:LISTENER WAS NOT GIVEN
Additional Information: The Oracle Database Extended agent logs are
written at a rapid pace and growing large very quickly.
The following error message repeats:

"ORA-12504: TNS:listener was not given".

APAR : IV30446
Abstract : ALERT LOG MESSAGE TEXT IS GARBLED
Additional Information: The Japanese part of the Oracle alert
log message is garbled in the Tivoli Enterprise Portal.
This garbled information might also occur in other multi-byte
character set (MBCS) locales, such as East Asian locales
such as Simplified Chinese or Korean.

APAR : IV31219
Abstract : RDB_SESSION_DETAILS WAREHOUSING ERROR 64
Additional Information: The historical data from the
attribute group RDB_Session_Details table cannot be
exported into the warehouse, because the value after UTF-8
conversion has a length greater than the column ACTION defined
width (32). Therefore, the row is not exported into
warehouse and the local historical data file size
increases continuously. This issue might also occur in other
East Asian locales such as Simplified Chinese or Korean.

APAR : IV36395
Abstract : KRZCLIENT FILE HANDLE LEAK ON WINDOWS
Additional Information: "Handle Count" on Windows operating systems
where the Oracle Extended Database agent is running is constantly
growing. When the agent is restarted, the krzclient.exe process
is using hundreds of file handles. The hundreds of file handles
increases to thousands of file handles in a few hours.

APAR : IV38555
Abstract : KRZCLIENT CRASHES AT TOPSQL
Additional Information: The Oracle Extended Database agent krzclient
process crashes and generates a core file when the process queries
RDB Top SQL attribute groups.

2.2 Defects
-----------

2.3 Enhancements:
-----------------

Enhancement: INTERNAL
Abstract: User interface configuration improvements.

- Simplifies the database connection configuration
with three connection types:
- Basic: Oracle database server host name (or IP address),
Listener port, and Oracle database service
name (or Oracle SID) must be specified.
- TNS: only the TNS Alias must be specified or selected
from the drop-down list that is discovered
from the ORACLE_HOME/network/admin/tnsnames.ora file.
- Advanced: only the Oracle connection string must be
specified to be compatible with an old
version of agent configuration.

- Supports the following connection string format:
"//hostname:port/ORACLE_SID".

- At least one database connection must be added during
the GUI configuration.

Enhancement: INTERNAL
Abstract: New environment variables that you can use to
customize the agent, such as KRZ_LISTENER_INCLUDE,
KRZ_LISTENER_EXCLUDE, KRZ_TNS_INCLUDE, KRZ_TNS_EXCLUDE,
KRZ_REDIRECT_TNS, KRZ_EVENT_LEVEL, and KRZ_SQL_PREFILTER.

- Provides the option to control whether the agent monitors
the specific Listener name and Oracle net service end points.
Both including and excluding filters are provided
(KRZ_LISTENER_INCLUDE,KRZ_LISTENER_EXCLUDE,
KRZ_TNS_INCLUDE, KRZ_TNS_EXCLUDE).

- Provides the option (KRZ_REDIRECT_TNS) to control whether
to redirect the value of the predefined TNS_ADMIN environment
variable to the agent log directory to disable the Oracle
sqlnet.log.

- Provides the option (KRZ_EVENT_LEVEL) to control which
level of agent event is sent. The level with a scope from high
to low is MISC, INFO, WARN, or ERROR.

- Provides the option (KRZ_SQL_PREFILTER) to control whether the
agent combines the filter information in the SQL query that is
sent to the Oracle database for the data processing.


Enhancement: INTERNAL
Abstract: Performance improvement for Top SQL monitoring.

Enhancement: INTERNAL
Abstract: Disk Space usage monitoring for Archive Destination
when Archive destination is defined in the Oracle database as
USE_DB_RECOVERY_FILE_DEST instead of a real physical directory.

Enhancement:
Abstract: More SQL statement samples in the krzcussql.properties
customized SQL template file for Oracle Jobs, Oracle Users,
and Oracle Queues monitoring.

A total of six sample SQL statements are in the default
customized SQL template file. To ensure the sample customized
SQL can be run successfully, the SELECT authority from
corresponding Oracle performance views must be granted to the
Oracle user of the agent in advance.

SQL_ID: mytablespace
SQL_TEXT: select * from dba_tables;
Oracle Views to be granted SELECT permission: dba_tables

SQL_ID: mytest
SQL_TEXT: SELECT SID,
SQL_CHILD_NUMBER FROM v$session
Oracle Views to be granted SELECT permission: v$session

SQL_ID: dba_objects
SQL_TEXT: SELECT * FROM dba_objects where rownum<200;
Oracle Views to be granted SELECT permission: dba_objects

SQL_ID: krz_jobs
SQL_TEXT: SELECT JOB JOBID,
LOG_USER LOGUSER,
PRIV_USER PRIVUSER,
SCHEMA_USER SCHEMAU,
TO_CHAR(LAST_DATE, 'YYYYMMDDHH24MISS') LASTSUCCD,
LAST_SEC LASTSUCCS,
NEXT_SEC NEXTSEC,
BROKEN JOBBROKEN,
INTERVAL INTERVALFN,
NVL(FAILURES,0) JOBFAILS,
(SYSDATE - NEXT_DATE) * 1440 NEXTSTART FROM SYS.DBA_JOBS;
Oracle Views to be granted SELECT permission: SYS.DBA_JOBS

SQL_ID: krz_users
SQL_TEXT: SELECT USERNAME,
USER_ID USERID,
ACCOUNT_STATUS STATUS,
LOCK_DATE LOCKDATE,
EXPIRY_DATE EXPIRYDATE,
DEFAULT_TABLESPACE DEFAULTTS,
TEMPORARY_TABLESPACE TEMPTS,
CREATED,
PROFILE FROM SYS.DBA_USERS;
Oracle Views to be granted SELECT permission: SYS.DBA_USERS

SQL_ID: krz_queues
SQL_TEXT: SELECT NAME,
QUEUE_TABLE QUEUETABLE,
QID QUEUEID,
QUEUE_TYPE TYPE,
MAX_RETRIES MAXRETRY,
RETRY_DELAY DELAY,
ENQUEUE_ENABLED ENQUEUE,
DEQUEUE_ENABLED DEQUEUE FROM SYS.DBA_QUEUES;
Oracle Views to be granted SELECT permission: SYS.DBA_QUEUES

2.4 Superseded fixes:
---------------------
6.3.1-TIV-ITM_KRZ-IF0001.
6.3.1-TIV-ITM_KRZ-IF0002.
6.3.1-TIV-ITM_EXT-FP0001.

2.5 APARS and defects included from superseded fixes
----------------------------------------------------
6.3.1-TIV-ITM_EXT-FP0001
----------------------------------------------------

APAR : IV22637
Abstract : AGENT CORE DUMP IF TNSPING IS NOT ACCESSABLE
Additional Information: A core dump occurs for the Oracle Database
Extended agent if the agent is configured with KRZ_ORACLE_HOME but,
$ORACLE_HOME/bin/tnsping file did not grant read+execute permission
to the agent OS user.

APAR : IV14403
Abstract : TABLESPACE PERCENT_FREE_TO_MAX CALCULATE WRONG
Additional Information: The Oracle Database Extended agent might calculate
the value of the Percent_Free_To_Max attribute incorrectly, because this value
should be greater than the value of the Percent_Free_To_Allocated attribute
in case the Oracle tablespace is set with Autoextend ON and
the maximum file size of the data file is greater than the allocated size
for the current datafile.

6.3.1-TIV-ITM_KRZ-IF0002
----------------------------------------------------

APAR: IV08618
Abstract : AGENT OCCUPY ORACLE SESSION MORE AND MORE
Additional Information: The Oracle Database Extended agent occupies the
Oracle sessions increasingly if the Oracle Database Extended agent user
does not have all authorities in the krzgrant.sql file.
This increase is caused by the agent not disconnecting from the Oracle
database, with an error message of "ORA-00942 Table or View does not
exist," and the agent trying to reconnect to the Oracle database with a new
session in each interval.

Defect: 167731
Abstract: agent configure still load based version xml file.
Additional Information: A new rz_dd_063100002.xml file is
installed in the itmhome\config directory by this patch,
where the base version of the rz_dd_063100000.xml file is installed.

The agent loads the configuration definition from the newly installed
rz_dd_063100002.xml file instead of the base version file.

Defect: 168694
Abstract: cannot ignore Oracle Listener/Net Service definition.
Additional Information: A new KRZ_LOAD_ORACLE_NET environment variable
is added so you can decide whether the Listener and Net Service End Points
from the files below are required for monitoring.

On UNIX and Linux systems:
$ORACLE_HOME/network/admin/listener.ora
$ORACLE_HOME/network/admin/tnsnames.ora
$TNS_ADMIN/network/admin/listener.ora
$TNS_ADMIN/network/admin/tnsnames.ora

On Windows systems:
%ORACLE_HOME%\network\admin\listener.ora
%ORACLE_HOME%\network\admin\tnsnames.ora
%TNS_ADMIN%\network\admin\listener.ora
%TNS_ADMIN%\network\admin\tnsnames.ora

If this new variable is set to 'false', the agent
does not load Listener or Net Service definitions from the files above,
even if the agent is configured with ORACLE HOME directory.

The agent still loads Listener or Net Service definition if the agent
is configured with a TNS PATH directory, which contains the
listener.ora or tnsnames.ora files.

Defect: 168966
Abstract: ASM disk/diskgroup display redundancy data.
Additional Information: In an Oracle RAC environment, if there is more
than one active ASM database instance, the agent queries back multiple
rows for each disk or disk group in all active instances.
If there is a situation in monitoring the attributes such as
KRZ_RDB_ASM_DISKGROUP.Free_Percent, there are
duplicated alerts by this type of situation.

The agent queries back only one record for each disk or disk group, and
avoids the duplicated alert for the situation.

The following attribute groups are impacted by this fix:

KRZ_RDB_ASM_DISKGROUP
KRZ_RDB_ASM_DISK
KRZ_ASM_DISK
KRZ_ASM_DISKGROUP

Defect: 168967
Abstract: FreeToMax return negative value.
Additional Information: The agent gets a negative value or over 100% on the
following attributes because the Maximum file size of an Oracle
data file is less than the real data file size:

KRZ_RDB_TABLESPACENORMAL_USAGE.Percentage_Free_To_Maximum
KRZ_RDB_TABLESPACENORMAL_USAGE.Percentage_Allocated
KRZ_RDB_TABLESPACETEMP_USAGE.Percentage_Allocated

Defect: 172392
Abstract: CANNOT MERGE CUSTOM SQL FOR BASETABLE
Additional Information: Use the KRZ_CUSTOM_SQLXML environment variable
to specify a customized
krzsql_custom.xml file where the new SQLInfo is defined and the agent
uses SQLInfo from the customized sql xml file instead of the
default krzsql.xml file. There is a new BaseTable tag in
the SQLInfo from version 6.3.1 and the new definition of
BaseTable cannot overwrite the default definition in the krzsql.xml file.

<SQLInfo ID=
...
</SQLText>
<BaseTable QueryName="SampleTable1" GrantName="SampleTable1"/>
</SQL>

If you do not grant all tables and views to the Oracle Database Extended agent
user in the krzgrant.sql file, error messages are displayed about ORA-00942
because the agent checks the Oracle authority on the QueryName element
while the agent runs continuously.

The error message about ORA-00942 can be avoided by using a customized
sql xml file and copying the SQLInfo definition from the default krzsql.xml
file and replacing the QueryName and GrantName with 'dual'.

<SQLInfo ID=
...
</SQLText>
<BaseTable QueryName="dual" GrantName="dual"/>
</SQL>

Defect: 172695
Abstract: KRZCLIENT LOG FILE LOOP AND OVERWRITE FIRST ONE
Additional Information: The RAS1 log file for the
krzstart or krzclient processes loops between 01, 02, and 03. The
first RAS1 log file is overwritten after the 3rd one
is full. This action disenables you to see information, such as
start time during agent startup.

After the application of the fix, in the following file examples
the RAS1 log message loops between the
"tivp56_rz_orcl_krzclient_4eef1fe8-02.log" file and the
"tivp56_rz_orcl_krzclient_4eef1fe8-03.log" file. The
"tivp56_rz_orcl_krzclient_4eef1fe8-01.log" file
is not overwritten.

#cd /opt/IBM/ITM/logs && ls *krzclient*.log
tivp56_rz_orcl_krzclient_4eef1fe8-01.log
tivp56_rz_orcl_krzclient_4eef1fe8-03.log
tivp56_rz_orcl_krzclient_4eef1fe8-02.log


6.3.1-TIV-ITM_KRZ-IF0001
----------------------------------------------------
APAR : IZ77624
Abstract : AGENT GETS ERROR ORA-01722 INVALID NUMBER ORACLE EXTENDED AGENT
6.3.0 ON DISTRIBUTED PLATFORM
Additional Information: The Oracle Database Extended agent returns no data
to the Tivoli Enterprise Portal because of ORA-01722.

APAR : IZ83799
Abstract : UNDO TABLESPACE %USED IS WRONG
Additional Information: The RDB_TablespaceNormal_Usage attribute group
PCIUSED (% Used) attribute is incorrect for UNDO tablespace.

APAR : IZ84903
Abstract : ITM RZ AGENT DOES NOT SUPPORT ORACLE 9I
Additional Information: The corrected prerequisite was republished
in the information center, see the "Supported applications" section at
http://publib.boulder.ibm.com/infocenter/tivihelp/v24r1/topic/com.ibm.itcama.doc_6.2.4/oracledatabaseextendedagent.html
Note: For Oracle Database Servers: Requires Version 10.1.0.1 or later.

APAR : IZ87013
Abstract : DBCONNECTION IN SUBNODE NOT TRUNCATED
Additional Information: Subnode ID as part of the subnode name must not be
longer then 25 characters, because of IBM Tivoli Monitoring limitations.
The entire subnode ID is truncated instead of the DBConnection part
being truncated.
Note: Subnode ID format: DBConnection-AgentInstance-Hostname.

The following troubleshooting Technote was published:
http://www-01.ibm.com/support/docview.wss?uid=swg21455513

APAR : IZ92572
Abstract : SITUATION DOES NOT FIRE WITH SCAN FILTER
Additional Information: The Oracle Database Extended agent gets an incorrect
filter operator of 'EQ' when a situation includes the SCAN filter with the 'NE'
operator.
The wrong operator causes the agent not to get the required data from the
Oracle database, and situations cannot be fired even if there are matched
data rows for this situation.

APAR : IZ95428
Abstract : KRZGRANT.SQL RETURNS ORA-01722 ERROR IN AN ORACLE ENVIRONMENT
USING A NON-STANDARD NLS_NUMERIC_CHARACHTERS STRING
Additional Information: Using a non-standard NLS_NUMERIC_CHARACTERS string
in the Oracle environment causes the problem.

APAR : IZ95117
Abstract : CANNOT MONITOR ORACLE TM LOCK
Additional Information: The agent can detect only the Transaction Lock
(type=TX) and cannot detect the DML or Table Lock(type=TM) in the Oracle
database.

APAR : IZ95898
Abstract : KRZCLIENT CORE DUMP AT ALERT LOG READING
Additional Information: krzclient causes a core dump if the alert log is archived
and newly generated alert log is the same size as the previous one.

Defect: 136835
Abstract: RDB Alert Log Details attribute group Message Timestamp attribute
shows an incorrect value for timestamp with timezone.

2.6 Enhancements included from superseded fixes
----------------------------------------------------
6.3.1-TIV-ITM_EXT-FP0001
----------------------------------------------------

Self-describing agent (SDA) application support installation is supported.
To take advantage of self-describing agent capability, your
Tivoli Management Services server components must be
Version 6.2.3 Fixpack 1 or later.

6.3.1-TIV-ITM_KRZ-IF0002
----------------------------------------------------
Enhancement: RFE
Headline: Improve Oracle Database Extended Agent not to contact ORCL instance
(SYS4472)
ID: 12585
Description:
Stop the behavior where the Oracle Extended Database agent tries to connect
against the Oracle default instance "ORCL" every time when checking
LISTENERs/TNS.
The Oracle Extended Database agent is hard-coded to try to connect against
an Oracle Instance "ORCL" that does not exist on most systems.
By default, the agent tries to connect every 5 minutes.
Therefore, the Oracle database creates "connect error 12514" Log Entries
each time, which fills up the log files.

Additional Information: The Oracle Extended Database agent uses the tnsping Oracle
utility tool in the ORACLE_HOME/bin path to verify the
Listener status if the agent is configured with KRZ_ORACLE_HOME. Or, the user
can use the new KRZ_DEFAULT_ORACLE_SERVICE_NAME environment variable to define
a valid service name for the Listener monitoring instead of the hard coded 'ORCL'.

Enhancement: RFE
Headline: RDB Disk Space: Oracle Archive directory is not monitored
correctly
ID: 13528
Description:
On SAP systems, the Oracle archive destination is defined as
log_archive_dest=<directory>/<file prefix>, which is not a
valid or existing directory. "Not Available" is displayed for % Free
attribute within RDB Disk Space attribute group.

Enhancement: RFE
Headline: Oracle RAC One Node Monitoring
ID: 13734
Description:
Oracle RAC One Node is a single instance database that is always online,
and it is a new option since Oracle 11.2.0.1. The Oracle
Extended Database agent is not able to monitor the RAC One Node correctly.
The agent displays Instance Status as "INACTIVE", and Instance Name as "*"
within the RDB Active Instance attribute group in a RAC One Node
cluster environment.

6.3.1-TIV-ITM_KRZ-IF0001
----------------------------------------------------

Enhancement: MR0329054818
Abstract: Add Host name, Instance name, and Database name to all Oracle
attribute groups for the ITCAM Extended Agent for Oracle Database version
6.3.
Additional Information: With this enhancement, the situation provides more
detailed information. The Oracle Database Extended agent also provides
this information in a combined string.
The IT administrator can set up the situation easily.

The following attribute groups were changed:
KRZ_DATABASE_INFORMATION
KRZ_AGENT_LISTENER
KRZ_RDB_ALERT_LOG_SUMMARY
KRZ_RDB_ALERT_LOG_DETAILS
KRZ_RDB_TABLESPACE_OVERVIEW
KRZ_RDB_TSFILE_METRIC
KRZ_RDB_SEGMENT_ALLOCATION
KRZ_RDB_TABLESPACETEMP_USAGE
KRZ_RDB_TABLESPACENORMAL_USAGE
KRZ_RDB_SEGMENT_STATISTICS
KRZ_RDB_DATAFILE_IO
KRZ_RDB_DATAFILE_OVERVIEW
KRZ_RDB_DATAFILE_COUNT
KRZ_RDB_TEMPFILE_OVERVIEW
KRZ_RDB_OBJECTS
KRZ_RDB_TABLES
KRZ_RDB_TABLE_SIZE
KRZ_RDB_TABLE_NUM_ROWS
KRZ_RDB_INDEXES
KRZ_RDB_INDEX_AVG_DATA_BLOCK
KRZ_RDB_INDEX_SIZE
KRZ_RDB_CLUSTERS
KRZ_RDB_CLUSTER_AVG_BLKS
KRZ_RDB_CLUSTER_SIZE
KRZ_DATABASE_OVERVIEW
KRZ_RDB_RESOURCE_LIMITATION
KRZ_RDB_SYS_STATISTICS
KRZ_RDB_SYS_METRIC_HISTORY
KRZ_RDB_DBA_HIST_SYSMETRIC_SUMMARY
KRZ_RDB_SYS_METRIC_GROWTH_PERCENT
KRZ_RDB_SYS_METRIC
KRZ_RDB_DATABASE_STATISTICS_SUMMARY
KRZ_RDB_DISK_SPACE
KRZ_RDB_PARAMETER
KRZ_RDB_OPTION
KRZ_RDB_SGA_OVERVIEW
KRZ_RDB_SGA_DETAIL
KRZ_RDB_SGA_SIZE
KRZ_RDB_PGA_OVERVIEW
KRZ_RDB_PGA_DETAIL
KRZ_RDB_PGA_TARGET_ADVICE
KRZ_RDB_DICTIONARY_CACHE_DETAIL
KRZ_RDB_DICTIONARY_CACHE_RATIO
KRZ_RDB_BUFFER_CACHE_DETAIL
KRZ_RDB_BUFFER_CACHE_RATIO
KRZ_RDB_LIBRARY_CACHE_DETAIL
KRZ_RDB_LIBRARY_CACHE_RATIO
KRZ_RDB_INSTANCES_ACTIVE_RATIO
KRZ_RDB_ACTIVE_INSTANCE
KRZ_RDB_INSTANCE_INFO
KRZ_RDB_INTER_CONNECTIONS
KRZ_RDB_GCS_CR_LATENCY
KRZ_RDB_GCS_CUR_BLOCK
KRZ_RDB_GCS_BLOCK_LOST
KRZ_RDB_GCS_MEM
KRZ_RDB_CACHE_FUSION_WRITE
KRZ_RDB_GES_MEM
KRZ_RDB_GES_ENQ_STAT
KRZ_RDB_GES_LATENCY
KRZ_RDB_GES_LOCK
KRZ_RDB_MSG_STAT
KRZ_RDB_SESSION_WAIT
KRZ_RDB_DBA_EXTENTS
KRZ_RDB_ASM_CLIENT
KRZ_RDB_ASM_DISK
KRZ_RDB_ASM_DISKGROUP
KRZ_RDB_ASM_DISKIOSTAT
KRZ_RDB_ASM_DISKGROUPIOSTAT
KRZ_RDB_ASM_TEMPLATE
KRZ_RDB_ARCHIVE_OVERVIEW
KRZ_RDB_ARCHIVE_DEST
KRZ_RDB_FLASH_RECOVERY_AREA_USAGE
KRZ_RDB_FLASHBACK_DATABASE_LOG
KRZ_RDB_FLASHBACK_DATABASE_LOGFILE
KRZ_RDB_RECOVERY_FILE_DEST
KRZ_RDB_SESSION_DETAILS
KRZ_RDB_SESSION_SUMMARY
KRZ_RDB_LOCK_STATISTICS
KRZ_RDB_LOCK_DETAIL
KRZ_RDB_LOCK_CONFLICT
KRZ_RDB_LATCH_STATISTICS
KRZ_RDB_LATCH_CHILDREN
KRZ_RDB_ROLLBACK_SEGMENT_SUMMARY
KRZ_RDB_ROLLBACK_SEGMENT_DETAIL
KRZ_RDB_UNDO_STATISTICS
KRZ_RDB_PROCESS_SUMMARY
KRZ_RDB_PROCESS_DETAIL
KRZ_RDB_BGPROCESS_STATS
KRZ_RDB_TOP_SQL
KRZ_RDB_DBA_HIST_SQL_PLAN
KRZ_RDB_CUSTOMIZED_SQL_SUMMARY
KRZ_RDB_CUSTOMIZED_SQL_DETAIL
KRZ_ASM_ALERT_LOG_SUMMARY
KRZ_ASM_ALERT_LOG_DETAILS
KRZ_ASM_INSTANCE
KRZ_ASM_PROCESS
KRZ_ASM_CLIENT
KRZ_ASM_BGPROCESS_STATS
KRZ_ASM_DISK
KRZ_ASM_DISKGROUP
KRZ_ASM_FILEINFO
KRZ_ASM_PARAMETER
KRZ_ASM_TEMPLATE
KRZ_ASM_DISKIOSTAT
KRZ_ASM_DISKGROUPIOSTAT
KRZ_ASM_OPERATION
KRZ_DG_DATABASE_OVERVIEW
KRZ_DG_STATS
KRZ_DG_STATUS
KRZ_DG_BGPROCESS_STATS
KRZ_DG_ALERT_LOG_SUMMARY
KRZ_DG_ALERT_LOG_DETAILS
KRZ_DG_LOG
KRZ_DG_LOGFILE
KRZ_DG_STANDBY_LOG
KRZ_DG_ARCHIVE_DEST_AND_STATUS
KRZ_DG_PARAMETER
KRZ_DG_DATAFILE
KRZ_DG_ARCHIVE_DEST
KRZ_DG_ARCHIVE_DEST_STATUS
KRZ_DG_ARCHIVED_LOG
KRZ_DG_LATEST_ARCHIVED_LOG
KRZ_DG_ARCHIVE_GAP
KRZ_DG_LOG_HISTORY
KRZ_DG_LOG_HISTORY_STAT
KRZ_DG_MANAGED_STANDBY
KRZ_DG_RECOVERY_PROGRESS
KRZ_DG_MANAGED_STANDBY_STAT
KRZ_DG_LOGSTDBY_PROCESS
KRZ_DG_LOGSTDBY_PROGRESS
KRZ_DG_LOGSTDBY_STATE
KRZ_DG_LOGSTDBY_STATS
KRZ_DG_LOGSTDBY_STATISTICS
KRZ_DG_LOGSTDBY_PROCESS_STAT
KRZ_DG_CUSTOMIZED_SQL_SUMMARY
KRZ_DG_CUSTOMIZED_SQL_DETAIL

Enhancement: INTERNAL
Abstract: Providing the option to control whether monitoring the listener
and Oracle net-service.
Additional Information: Listener monitoring and net-service monitoring
are started by default. If you do not want this
monitoring, you can turn off the related monitoring functions.
The KRZ_LISTENER_PING_INTERVAL and KRZ_TNS_PING_INTERVAL environment
variables control the interval. If the value is less than 1, the agent
exits the thread and stops checking.

Enhancement: INTERNAL
Abstract: Adding the delta value for system level statistics data.
Additional Information: With this enhancement, you can monitor the delta
changed value for a specific metric.

Enhancement: INTERNAL
Abstract: Enhancement at subnode name. You can control how the subnode
name is presented in IBM Tivoli Monitoring.
Additional Information: This enhancement also protected an existing
problem: "pure event cannot fire when subnode name length is 32 characters",
which is a known APAR at ITM622FP4.

Present the subnode name as
"RZ:<database connection name>-<agent instance name>-<host name>:RDB".

The KRZ_SUBNODE_INCLUDING_AGENTNAME environment variable controls whether
the agent instance name is added to the subnode name.
The default value is YES.

The KRZ_SUBNODE_INCLUDING_HOSTNAME environment variable controls whether
the host name is added to the subnode name.
The default value is YES.

The KRZ_MAX_SUBNODE_NAME_LENGTH environment variable controls the
maximum subnode name length.
The default value is 24.


3.0 Architecture and prerequisites
==================================

This fix is supported on all operating systems listed in
the IBM Tivoli Composite Application Manager Extended Agent for Oracle
Database V6.3.1 Fixpack 2 User's Guide.

The following website contains the latest certification information:
Software Product Compatibility Reports (SPCR)
http://publib.boulder.ibm.com/infocenter/prodguid/v1r0/clarity/index.html

3.1 Upgrade prerequisites for this fix
-----------------------------
There is no prerequisite level for this fix.

Because this fix is cumulative, it can be installed on any fix level
for the version, release, and modification level above the prerequisite.


4.0 Image directory contents
============================
Physical Part Number: ITCAM Apps Ag for Oracle CBAQ2ML
Electronic Part Number: ITCAM Apps Ag for Oracle CIL7KML

5.0 Installation instructions
=============================

When this fix pack image is downloaded from the IBM Software Support website,
these rules apply:
* You can locally install the fix pack only on a system that already hosts a
licensed copy of IBM Tivoli Monitoring for Databases. Similar to other
upgrade software, the fix pack image also can be locally installed on a
computer where the product software is not already installed.
* You can populate agents to the depot.
* You can remotely deploy agents to existing or new installations.

When this fix pack image is downloaded from Passport Advantage, these rules
apply:
* You can locally install the fix pack on a system that already hosts a
licensed copy of IBM Tivoli Monitoring for Databases.
* You can locally install the fix pack on a system that does not already host
a licensed copy of IBM Tivoli Monitoring for Databases.
* You can populate agents to the depot.
* You can remotely deploy agents to existing or new installations.


5.1 Before installing the fix
-----------------------------
- The prerequisites listed under section 3.1 entitled 'Upgrade prerequisites
for this fix' must be installed when upgrading.

- For the purpose of this README, the <CANDLEHOME> symbol is the
IBM Tivoli Monitoring installation directory. The default value
for CANDLEHOME is '/opt/IBM/ITM' on UNIX systems and 'C:\ibm\itm'
on Windows systems.

- If you have customized the Extended Monitoring Agent for Oracle
rz.ini file on a UNIX system, back up this file before installing the fix.
After this fix is installed, edit the latest rz.ini file and merge
your modifications into this rz.ini file.


5.2 Local agent install and update (including silent installation)
--------------------------------------------------------------
1. Transfer the appropriate archive file to a
temporary directory on the system that contains the agent code to be
updated. For the purpose of this README, the <TEMP> symbol represents
the fully qualified path to this directory.
Note: On Windows systems, this path includes the drive letter.

2. Expand the archive file by using the tar command on UNIX systems or
an unzip utility on Windows systems. This expansion creates a directory
structure that contains fixes for all supported platforms.

3. Use the procedures in the "IBM Tivoli Monitoring, Version 6.2.2
Installation and Setup Guide" to install the agent.

4. Use the procedures in the specific database monitoring agent user's
guide to complete the installation and configuration.


5.3 Remote agent update and installation
----------------------------------------
1. Make sure that all potential targets of the remote deployment are
running version 6.2.2 Fix Pack 2 or later of the OS monitoring
agent.
2. Transfer the appropriate archive file to
a temporary directory on the Tivoli Enterprise
Monitoring Server system. For the purpose of this README, the
<TEMP> symbol represents the fully qualified path to this directory.
Note: On Windows systems, this path includes the drive letter.

3. Expand the archive file by using the tar command on UNIX systems or an
unzip utility on Windows systems. This expansion creates a directory structure
that contains fixes for all supported operating systems.

4. To add the fix pack bundles to the remote deployment depot, use the
tacmd addBundles command found in $CANDLEHOME/bin on UNIX systems
or in %CANDLE_HOME%\bin on Windows systems. For more information
about the tacmd addBundles command, see the "IBM Tivoli
Monitoring Command Reference."

On a UNIX system: if the fix was expanded to
<TEMP>/<PARTNUMBER>:
> $CANDLEHOME/bin/tacmd addBundles -i <TEMP>/<PARTNUMBER>/WINDOWS/Deploy -t rz -f
> $CANDLEHOME/bin/tacmd addBundles -i <TEMP>/<PARTNUMBER>/WIA64/Deploy -t rz -f
> $CANDLEHOME/bin/tacmd addBundles -i <TEMP>/<PARTNUMBER>/unix -t rz -f

where:
-i is the directory that contains the deployment bundles to be
added to the depot.
-t is the product code of the product to add, in this case rz
represents the Extended Monitoring Agent for Oracle Database
-f performs the actions without asking for confirmation

On a Windows system: if the fix was expanded to
<TEMP>/<PARTNUMBER>:
> %CANDLE_HOME%\bin\tacmd addBundles -i <TEMP>\<PARTNUMBER>\WINDOWS\Deploy -t rz -f
> %CANDLE_HOME%\bin\tacmd addBundles -i <TEMP>\<PARTNUMBER>\WIA64\Deploy -t rz -f
> %CANDLE_HOME%\bin\tacmd addBundles -i <TEMP>\<PARTNUMBER>\unix -t rz -f

where:
-i is the directory that contains the deployment bundles to be
added to the depot.
-t is the product code of the product to add, in this case rz
represents the Extended Monitoring Agent for Oracle Database
-f performs the actions without asking for confirmation

5. To login to the Tivoli Enterprise Monitoring server and deploy
the fix to the appropriate nodes where the agent is running, use
the following tacmd commands. For more information about the
tacmd login and updateAgent commands, see the "IBM Tivoli
Monitoring Command Reference."

On a UNIX system:
> $CANDLEHOME/bin/tacmd login -s <server>
-u <itmuser>
-p <password>

> $CANDLEHOME/bin/tacmd listSystems

The output shows the managed system name for the OS agent on the
remote system to be updated. Use this value as the target of the
tacmd updateAgent command.

> $CANDLEHOME/bin/tacmd updateAgent -t rz
-n <Managed system name>
-v 063102000

On a Windows system:
> %CANDLE_HOME%\bin\tacmd login -s <server>
-u <itmuser>
-p <password>

> %CANDLE_HOME%/bin/tacmd listSystems

The output shows the managed system name for the OS agent on the
remote system to be updated. Use this value as the target of the
tacmd updateAgent command.

> %CANDLE_HOME%\bin\tacmd updateAgent -t RZ
-n <Managed system name>
-v 063102000
Note:
- The component (-t) for the updateAgent command is specified as
two characters (rz), not three characters (krz).
- The node (-n) for the updateAgent command is the managed system
name of the operating system (OS) agent to be updated. The
target node for an updateAgent command is always an OS agent.

- The following error might occur with the updateAgent command on
Linux and UNIX endpoints; however, the agent is successfully
updated. This error does not occur on Windows endpoints.

KUICUA011I: Updating the rz agents.
KUICUA015E: The updateAgent command did not complete because an
error occurred. Refer to the following error
returned from the server:

The monitoring server encountered an error while updating the
managed system type.

The kuiras1.log file might provide more information about
this error. If you require further assistance to resolve the
error, contact IBM Software Support.

A timeout occurred while waiting for an agent task to complete on
{node name}.

The agent task did not complete within the configured
time out period.

Verify that the target system is online. Increase the
time out value and try the operation again.



5.4 Agent support update
------------------------
Fix 6.3.1-TIV-ITM_EXT-FP0001 included changes to the agent
support files that must be installed. If you have already
installed these updates, there are no additional installation
steps. Otherwise, use the following steps to update the Tivoli
Enterprise Monitoring Server, Tivoli Enterprise Portal Server, or
Tivoli Enterprise Portal Desktop.

1. Transfer the appropriate archive file
to the IBM Tivoli Enterprise Monitoring Servers, IBM Tivoli
Enterprise Portal Servers, or Tivoli Enterprise Portal desktops.

2. Expand the archive file using the tar command on UNIX systems or an
unzip utility on Windows systems.

3. Use the procedures in the "IBM Tivoli Monitoring, Version 6.2.2
Installation and Setup Guide."

4. Use the procedures in the specific database monitoring agent user's
guide to complete the installation and configuration.

5. If the Tivoli Enterprise Monitoring Server being updated is remote (not a
Hub Tivoli Enterprise Monitoring Server) then restart the Tivoli Enterprise
Monitoring Server.


6.0 Additional installation information
========================================

6.1 Verifying the update
------------------------
1. To verify that the agent was updated correctly, use the tacmd command to
view the current version of the agent. You are required to log in
to a Tivoli Enterprise Monitoring Server before you view the agent
version.

For example:
On a UNIX system, where $CANDLEHOME is the IBM Tivoli Monitoring
installation directory, the default location is '/opt/IBM/ITM'.

> $CANDLEHOME/bin/tacmd login -s <server>
-u <itmuser>
-p <password>
> $CANDLEHOME/bin/tacmd listSystems -t rz

On a Windows system, where %CANDLE_HOME% is the IBM Tivoli
Monitoring installation directory, the default location
is 'C:\ibm\itm'.

> %CANDLE_HOME%\bin\tacmd login -s <server>
-u <itmuser>
-p <password>
> %CANDLE_HOME%\bin\tacmd listSystems -t RZ

When the agent update is successful, the agent version is
06.31.02.00

Note:
The component (-t) for the tacmd listSystems command is specified
as two characters (RZ), not three characters (KRZ).

2. To verify that the agent support files were updated correctly, use
the kincinfo command on Windows systems or the cinfo command on Linux
or UNIX systems to dump the component versions on each of the
following systems: Tivoli Enterprise Portal Server, Tivoli Enterprise Monitoring
Server, and Tivoli Enterprise Portal desktop.

Note:
The kincinfo and cinfo command output examples in the sections
that follow show all components that can be updated. If you do
not have all of these components installed, components that
are not installed are not displayed in the output of these commands.


Sample output for the cinfo command on a UNIX or Linux systems
--------------------------------------------------------------
To validate that all components are installed, run the
following command:

./cinfo

This command provides four options on the CINFO menu shown in
the example that follows. Choose the option that meets your requirements.

*********** Thu Aug 15 15:02:01 HKT 2013 ******************
User : root Group: root
Host name : myaixsys Installer Lvl:06.22.02.00
CandleHome: /opt/IBM/ITM
***********************************************************

-- CINFO Menu --
1) Show products installed in this CandleHome
2) Show which products are currently running
3) Show configuration settings
4) Show installed CD release versions
X) Exit CINFO
1

*********** Thu Aug 15 15:02:01 HKT 2013 ******************
User : root Group: root bin daemon sys adm disk wheel
Host name : myaixsys Installer Lvl:06.22.02.00
CandleHome: /opt/IBM/ITM
***********************************************************
...Product inventory


Unix/Linux: IBM Tivoli Enterprise Server Portal Server / IBM Tivoli
Enterprise Portal Desktop, cinfo output:

rz ITCAM Extended Agent for Oracle Database
li6263 Version: 06.31.02.00

Unix/Linux: Extended Agent for Oracle Database


rz ITCAM Extended Agent for Oracle Database
li6263 Version: 06.31.02.00

Sample output of the kincinfo command on a Windows System
----------------------------------------------------------
To validate that all components have been installed, run
the following command:

For example:
> %CANDLE_HOME%\InstallITM\kincinfo -i

>kincinfo -i

kincinfo output:

Windows: Tivoli Enterprise Server Portal Server / Tivoli
Enterprise Portal Desktop

RZ ITCAM Extended Agent for Oracle Database
WINNT Version: 06.31.02.00 Build: 201308012231

RZ ITCAM Extended Agent for Oracle Database
WINNT Version: 06.31.02.00 Build: 201308012231

RZ ITCAM Extended Agent for Oracle Database
WINNT Version: 06.31.02.00 Build: 201308012231

RZ ITCAM Extended Agent for Oracle Database
WINNT Version: 06.31.02.00 Build: 201308012231

Windows: Extended Agent for Oracle Database:

RZ ITCAM Extended Agent for Oracle Database
WINNT Version: 06.31.02.00 Build: 201308012231


7.0 Known problems and workarounds
==================================
- Problem: Oracle agent unable to logon to oracle database, due to
invalid password, after remote deploying or remote
configuring, on a windows x64 system.
defect: 119668
Workaround: Upgrade the Windows OS agent to version 6.22 fp2.

- Problem: Oracle agent running processes not stopped, after removing
the oracle agent instance from the tivoli enterprise
monitoring services management window, on a UNIX system.
defect: 120864
Workaround: Stop the Oracle Database Extended agent instance before removing it.

- Problem: A connection of oracle agent not displayed on TEP, if the
connection name is too long.
defect: 121337
Workaround: Assign a short name to the Oracle Database Extended agent instance and
connection in the instance to avoid truncating the unique name(connection
name + krz instance name + host name).

- Problem: Unable to remote deploy oracle agent to a windows IA64
system, if no WINNT platform bundle exists in the TEMS
repository.
defect: 122095
Workaround: Add the 3 Windows operating system bundles (WINNT, WIX64, WIA64) to
The Tivoli Enterprise Monitoring Server repository before remotely deploying
on a Windows IA64 system.

- Problem: Alert logs is not displaying customized message ID.
- Workaround: If you must monitor custom message IDs in Alert logs,
message IDs must be specified in the
installdir/config/<host>_rz_<inst>_msgid.properties file
and you must restart the agent to display it in the portal.

- Problem: default value for "Last Execution Error Code" is displaying as 0
in workspace "Customized SQLs->RDB Customized SQL Summary status".
- Workaround: The customized SQLs are not executed during agent startup
by default. These SQLs are executed only if there is a situation or
you click "Link to SQL Result" in the workspace "Customized SQLs ->
RDB Customized SQL Summary Definition".

- Problem: agent environment variable or error trace level do not take
effective.
- Workaround: If you must edit an environment variable such as setting a new
error trace level, the new variable must be modified in both the
installdir/config/rz.ini file and the installdir/config/rz_<instance>.config
file, and you must restart agent.

- Problem: agent report ORA-00942 error message in the log files like
following. Oracle Database Server is 10.1.0.2.
ERROR at line 1:
ORA-00942: table or view does not exist
ORA-06512: at line 128
- Workaround: Run the krzgrant.sql script in 6.3.1-TIV-ITM_KRZ-FP0001 to resolve
this error.

- Problem: tacmd listsystems command does not display proper version
for offline agent's subnode.
For subnode e.g. RZ:HOSTNAME-INSTANCE-CONNECTION:RDB, version is
displaying as 06.31.01.XX, Status is 'N'.
- Workaround: Check the version from the root node of the agent to get the correct
version, for example, for INSTANCE:HOSTNAME:RZ, the version is displayed as
06.31.01.00.

- Problem: After upgrading to Fix Pack 2 and configuring the Oracle
Database Extended agent silently, the agent cannot
connect to the database, which is specified by the
KRZ_CONN_STR parameter in the silent configuration
response file.
- Workaround: The KRZ_CONN_TYPE parameter specifies the connection
type. The Advanced type is required to configure a
database connection with the KRZ_CONN_STR
connection string. For example:

KRZ_CONN_STR.connection=//host:port/service
KRZ_CONN_TYPE.connection=Advanced

To solve the problem, update the silent configuration
response file to specify the KRZ_CONN_TYPE parameter and
reconfigure the agent instance silently. For example:

# cat /tmp/silent_config.txt
############ PRIMARY TEMS CONFIGURATION ############
CMSCONNECT=YES
HOSTNAME=tivp055.cn.ibm.com
NETWORKPROTOCOL=ip.pipe
IPPIPEPORTNUMBER=1918
############ Database connection config ############
INSTANCE=inst1
KRZ_CONN_USERID=tivoli
KRZ_CONN_PASSWORD=password
KRZ_INSTANT_CLIENT_LIBPATH=/opt/IBM/oci
KRZ_DYNAMIC_LISTENER=FALSE
###
KRZ_CONN_STR.asm=//tivx010:1521/+ASM
KRZ_CONN_USERID.asm=sys
KRZ_CONN_PASSWORD.asm=oracle
KRZ_CONN_MODE.asm=SYSDBA
KRZ_CONN_TYPE.asm=Advanced

# /opt/IBM/ITM/bin/itmcmd config -A -o inst1
-p /tmp/silent_config.txt rz

- Problem: The Oracle Database Extended agent reports the error
message ORA-03135: connection lost contact and the
subnode that represents the monitored database
connection is still offline in the Tivoli Enterprise Portal
after the monitored database recovers to active status..
- Workaround: The Oracle Database might disconnect all active connections
and reject new connections under certain circumstances.
An example is when resources such as CPU are exhausted
in the Oracle Server.
To solve the problem, ensure that the Oracle Database
and Oracle Listener are open to the new connection and
restart the agent instance.

- Problem: After you upgrade to V6.3.1 Fix Pack 2, the subnode ID
is truncated from 25 characters to 24 characters. The
original subnode with 25 characters changes to gray, and
the new subnode with 24 characters is displayed.
- Workaround: This behavior is expected. In 6.3.1-TIV-ITM_KRZ-IF0001
or later versions, the agent changed the maximum
subnode ID from 25 characters to 24 characters to avoid
the following problem, which is a known APAR for IBM
Tivoli Monitoring V6.2.2 Fix Pack 4: "pure event cannot
fire when subnode name length is equal to or greater
than 32 characters."
To avoid truncation of the subnode from 25 characters to
24 characters, use one of the following two options:

Option 1: Set the following variable to a value of 25 in
the agent environment variable files:
KRZ_MAX_SUBNODE_ID_LENGTH=25. After you upgrade,
the default value is 24.
Note:
- On UNIX or Linux operating systems, the
environment variable files include
installdir/config/rz.ini and installdir/config/
rz_instance.config.
- On Windows operating systems, the environment
variable files include installdir\TMAITM6\
KRZENV_instance (32-bit) or installdir\
TMAITM6_X64\KRZENV_instance (64-bit).

Option 2: Reduce the length of the database
connection name, agent instance name, or host name;
or, see Changing default naming conventions for
database connections in the Installation and
Configuration Guide for the Oracle Database Extended
agent for information.

8.0 Additional product information
==================================
None.


9.0 Notices
===========
This information was developed for products and services offered
in the United States. IBM may not offer the products, services, or
features discussed in this document in other countries. Consult
your local IBM representative for information on the products and
services currently available in your area. Any reference to an
IBM product, program, or service is not intended to state or imply
that only that IBM product, program, or service may be used. Any
functionally equivalent product,program, or service that does not
infringe any IBM intellectual property right may be used instead.
However, it is the user's responsibility to evaluate and verify
the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject
matter described in this document. The furnishing of this document
does not grant you any license to these patents. You can send license
inquiries, in writing, to:

IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
U.S.A.

The following paragraph does not apply to the United Kingdom or
any other country where such provisions are inconsistent with local law:

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION
"AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Some states do not allow disclaimer of express or implied warranties
in certain transactions, therefore, this statement may not apply to you.

Trademarks and service marks
------------------------------------
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business
Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be
trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at
"Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.

Linux is a registered trademark of Linus Torvalds in the United States,
other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of
Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States
and other countries.

Other company, product, or service names may be trademarks or service
marks of others.

On
[{"DNLabel":"6.3.1-TIV-ITM_EXT-FP0002.README","DNDate":"30 Aug 2013","DNLang":"English","DNSize":"999999","DNPlat":{"label":"AIX","code":"PF002"},"DNURL":"http://www.ibm.com/support/fixcentral/quickorder?product=ibm%2FTivoli%2FTivoli+Composite+Application+Manager+for+Applications&fixids=6.3.1-TIV-ITM_EXT-FP0002&source=SAR ","DNURL_FTP":" ","DDURL":" "}]
[{"Product":{"code":"SS3JRN","label":"Tivoli Composite Application Manager for Applications"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"ITCAM Extended Agent for Oracle Database 5724V09OR v6.3","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"6.3.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Problems (APARS) fixed
IV38555;IV36395;IV31219;IV30446;IV26915;IV24938;IV22637;IV14403;IV08618;IZ95898;IZ95428;IZ95117;IZ92572;IZ87013;IZ84903;IZ83799;IZ77624

Document Information

Modified date:
13 July 2018

UID

swg24035571