by Vedant Chandratre

Hadoop is an open source frame work that is designed for distributed storage and processing of very large data sets called as big data. Hadoop provides massive storage for any kind of data, huge processing power, and the capability to handle virtually limitless tasks simultaneously. Typically, Hadoop administrators face the following issues that are related to Hadoop clusters:

• DataNode slow downs
• The amount of memory used exceeds the expected threshold
• The time required for RPC to complete exceeds the expected threshold
• Poor allocation of resources

​To ensure high availability and optimum performance of Hadoop clusters, IBM has introduced the monitoring agent for Hadoop. The Hadoop agent monitors the health and performance of Hadoop clusters and its different resources. The agent alerts administrators when the performance of any of the Hadoop resources fails to meet the expected criteria.

The agent provides the following details about a Hadoop cluster:​

• The status of Java virtual machines, remote procedure calls, and file system
• Information about process block operations
• Total number of files and file operations that occur in the cluster
• The status of applications in the cluster
• The data that is written to a DataNode and the data that is read from a DataNode
• The amount of memory that is used by all the nodes in the cluster
• The memory heap and non-heap details
• The status of a process thread
• The error log information

​Administrators can analyze the data that the agent collects, and can take corrective actions to resolve or prevent issues in Hadoop clusters.

To know more about the capabilities of the agent, go to the IBM Performance Management Knowledge Center.

If you have any further questions on monitoring agent for Hadoop, please add a comment to this blog post.