Configuring Hadoop monitoring

Edit online

You must configure the Monitoring Agent for Hadoop so that the agent can collect data of a Hadoop cluster that it monitors. The agent can monitor a single node Hadoop cluster and a multi-node Hadoop cluster.

Before you begin

Review the hardware and software prerequisites. For the up-to-date system requirement information, see the Software Product Compatibility Reports (SPCR) for the Hadoop agent.

Ensure that the following hosts can be resolved from the computer where the Hadoop agent is installed:
  • All the Hadoop hosts that you want to configure, such as NameNode, ResourceManager, and so on
  • Hadoop hosts with only the NodeManager role
For example, you can complete these steps to resolve hosts:
  • Add the IP address, host name, and fully qualified domain name of all the Hadoop hosts to the hosts file that is available at the following path:
    • WindowsC:\Windows\System32\drivers\etc\hosts
    • AIXLinux/etc/hosts
  • Add the computer where the Hadoop agent is installed in the same domain as that of Hadoop hosts.
Remember: To monitor a Hadoop cluster that is secured with Kerberos SPNEGO-based authentication, ensure that all the hosts can be resolved from the computer where the Hadoop agent is installed.

About this task

The Hadoop agent is a single instance agent. You must configure the agent manually after it is installed. The Hadoop agent can be configured on Windows, Linux®, and AIX® systems.
Remember:
  • For a single node Hadoop cluster, the same node performs all the roles, such as NameNode, ResourceManager, and secondary NameNode according to configuration of the Hadoop cluster. However, for a multi-node Hadoop cluster, different Hadoop nodes perform these roles.
  • When you configure the agent, the agent automatically detects DataNodes and NodeManagers in the Hadoop cluster that is being monitored.
  • Restart the agent if additional namenodes are configured in the HA enabled Hadoop cluster after the agent is started.
  • If the Hadoop flavor configured with Hadoop Agent is Cloudera, the name and displayname of the Hadoop cluster must not contain the character %.
Note: When the Hadoop agent is upgraded to the latest version such as 08.21.09.00, the agent must be reconfigured.
When you upgrade from the socket - based agent (8.1.2 Fix Pack 2, or earlier) to the REST API - based agent (8.1.3, or later), complete the configuration steps that are specified in the subsequent topics. However, ensure that you specify the host names according to the following guidelines when you configure the agent.
  • The host name of various daemon processes (NameNode, ResourceManger, and so on) that you specify must be the same (case and format) as the host names that are configured for the socket - based agent.
  • The fully qualified domain name (FQDN) must be used when you specify a host name. For example, hos1.ibm.com. If the length of the FQDN exceeds 25 characters, specify only the short host name without the domain name. For example, if the FQDN of a host is myhadoopclustersetupnode.ibm.com, the short host name is myhadoopclustersetupnode.
After you configure the agent that is upgraded, and view data in the Cloud APM console, revert the changes that were made in the hadoop-metrics2.properties file for the Hadoop agent. For details, see Upgrading your agents.

On Windows systems, you can run the Hadoop agent with a non-administrator user. However, such user requires a specific permission to view data in the dashboards. For information about how to grant this permission, see Granting permission to non-admin users.

The product version and the agent version often differ. The directions here are for the most current release of this agent. For information about how to check the version of an agent in your environment, see Agent version command. For detailed information about the agent version list and what's new for each version, see the Change history.