IBM Operations Analytics - Log Analysis, Version 1.3.1Configuring Hadoop for long term data storage
Before you can use Hadoop for long term data storage, you must configure the integration with IBM® Operations Analytics - Log Analysis.
Why should I use Hadoop?
Hadoop offers a more efficient method for long term data storage that you can use to store long term data from annotated log files. The integration with IBM Operations Analytics - Log Analysis is facilitated by a service that is bundled with IBM Operations Analytics - Log Analysis 1.3.1 and ensures that you can continue to search this data without need to store the data in the main database.
Hadoop integrations
- IBM InfoSphere® BigInsights® Hadoop
- IBM InfoSphere BigInsights delivers a rich set of advanced analytics capabilities that allows enterprises to analyze massive volumes of structured and unstructured data in its native format. For more information, see the IBM InfoSphere BigInsights documentation at http://www-01.ibm.com/support/knowledgecenter/SSPT3X/SSPT3X_welcome.html
- Cloudera Hadoop
- Cloudera provides a scalable, flexible, integrated platform that makes it easy to manage rapidly increasing volumes and varieties of data in your enterprise. For more information, see the Cloudera documentation at http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/introduction.html
Architecture
IBM Operations Analytics - Log Analysis stores the log data on to the Hadoop cluster and allows users to search data that is stored in Hadoop. IBM Operations Analytics - Log Analysis, with the help of the IBM Operations Analytics - Log Analysis service that is installed on each datanode, writes the data to the Hadoop cluster. The data is written in the avro object container file format. For more information about object container files, see http://avro.apache.org/docs/1.7.4/spec.html#Object+Container+Files. Data is then written to each datanode where the service is installed. You can use IBM Operations Analytics - Log Analysis to search this data.
You can also run IBM Operations Analytics - Log Analysis searches on the data stored on the Hadoop cluster.
The following graphic displays an overview of the service architecture:
