Viewing health notifications with JMX utilities

If a failure or recovery of data occurs on a server, then you can be automatically notified. You can view the notification to determine the required action, if any.

About this task

Use JMX utilities as a preventive action and for failure detection. For example, you might receive a message that indicates a process that did not complete correctly. Although a failure has not occurred yet, this automated message gives you an opportunity to act before the server goes down.

Procedure

Decide which utility you want to use.
For example, JMX utilities include the following tools:
- A stand-alone Java client, which is written to listen to the notifications
- The JConsole.exe JMX listener that is provided by the JDK
- IBM Tivoli Monitoring
- Any other JMX listening client
Determine how you want to connect to the catalog server.
- Connect with the management concentrator, which is hosted on the catalog server. The management concentrator collects all notifications from the container servers. It reduces the number of connections that need to be maintained from JMX listeners.
  The management concentrator is available on the catalog servers from the ManagementConcentratorMBean MBean. It can be accessed from the catalog server JMXService port. Use the following code example to connect to the catalog server with the management concentrator:
```
String serviceURL = "service:jmx:rmi:///jndi/rmi://" + hostName + ":" + JMXPort + "/objectgrid/MBeanServer";
JMXServiceURL jmxUrl = new JMXServiceURL(serviceURL);
JMXConnector jmxCon = JMXConnectorFactory.connect(jmxUrl);
MBeanServerConnection catalogServerConnection = jmxCon.getMBeanServerConnection();

Set<ObjectName> placementSet = catalogServerConnection.queryNames(new ObjectName(
"com.ibm.websphere.objectgrid" + ":*,type=ManagementConcentrator"), null);
ObjectName managementConcentrator = (ObjectName) placementSet.iterator().next();
NotificationFilterSupport filterSupport = null;
catalogServerConnection.addNotificationListener(managementConcentrator, this, filterSupport, null);
```
- Directly connect to all servers in environment.
  This option requires keeping JMX connections for all servers in the environment. If you have a large topology and aggressive timeout values, then listening directly to the servers instead of through the management concentrator can be worth the added maintenance of the JMX connections. For example, if you have a 5-second timeout on client calls, then you might prefer to listen directly to the container servers for their notifications.
- Use both the management concentrator and a direct connection to the server. Although you might choose to use both, be prepared to received duplicate notifications.
Decide which notifications you want to filter.

There are several filters available on the catalog server and container servers. To review the available notifications, see the API documentation: DynamicServerMBean and the API documentation: PlacementServiceMBean .

For example, in a multimaster replication (MMR) environment, it is useful to listen to the DynamicServerMBean notifications, com.ibm.websphere.objectgrid.replication.error and com.ibm.websphere.objectgrid.replication.recovery. These notifications report when a primary shard fails to replicate with its foreign primary and if it recovers replicating to the same location.
Determine which steps to take for different notifications.
To view a complete list of JMX notifications, see Package com.ibm.websphere.objectgrid.management in the API documentation.
Optional: Limit the number of notifications that are received by the management concentrator by setting the include or exclude policy for the management concentrator.
1. Set the includeManagementConcentratorNotifications and excludeManagementConcentratorNotifications properties in the catalog server properties file.
2. Use the xscmd -c setManagementConcentratorState include and exclude options.