Data type configuration

You can include custom data type definitions in your custom Insight Pack.

You can create data type configurations for each of the following entities:

Collections

You use a Collection to group log data from different data sources that have the same Source Type. The Collection definition depends on the Source Type definition that specifies how the IBM® Operations Analytics - Log Analysis Server splits, annotates, and indexes the incoming data records. You must define values for the following properties in the Collection definition:

Name: Specify a unique name that is used to identify the Collection.
Source Type: Specify the name of the Source Type that is associated with the log records in the Collection.

Source Types

A Source Type defines how data of a particular type is split, annotated, and indexed by IBM Operations Analytics - Log Analysis.

The Source Type specifies the Rule Sets and, if you want to implement custom processing, the File Sets that the IBM Operations Analytics - Log Analysis Server uses to split and annotate the log records for the particular data Source Type. The Source Type specifies the index configuration settings that the IBM Operations Analytics - Log Analysis uses to index the log records for the particular data Source Type.

You must define values for the following properties in the Source Type definition:

Name: Specify a unique name that is used to identify the Source Type.
Enable splitter: Select this flag to enable the splitter function that splits the log records during processing.
Splitter Rule Set name: Specify the name of the Annotation Query Language (AQL) rule set that governs how log records are split.
Splitter File Set name: Specify the name of a file that you created that contains custom splitter logic that you defined, for example Java™ or Python script, that governs how log records are split. This is an alternative to the Rule Sets.
Enable annotator: Select this flag to enable the annotator function that annotates the log records during processing.
Annotator Rule Set name: Specify the name of AQL rule set used to perform annotator function.
Annotator File Set name: Specify the name of a file that you created that contains custom annotator logic that you defined, for example Java Archive (JAR) or Python script, that governs how log records are annotated. This is an alternative to the Rule Sets.
Deliver data on annotator execution failure: Set this indicator to enable indexing even when the annotation fails. By default, indexing is stopped if the annotation fails.
Index configuration: Specify the name of index configuration JSON file that you use in your custom Insight Pack.

Rule Sets

A Rule Set is a collection of files that contain rules that are written in the Annotation Query Language (AQL). IBM Operations Analytics - Log Analysis uses the AQL rules to split logical log records according to a known boundary or to extract the data from fields in log records that contain structured or semi-structured data.

You must define the following properties in the Rule Set definition:

Name: Specify a unique name that is used to identify the Rule Set.
Type: Specify whether you want the Rule Set to split or annotate the log records.
Rule file directory: Specify the paths for the directories that contain the AQL rule files that the Rule Set uses. The paths must be relative to the src-files directory path that is defined in your custom Insight Pack. For example, extractors/ruleset/common;extractors/ruleset/splitterSystemOut.

File Sets

A File Set is a collection of files that contain the custom logic that you defined to split or annotate log data. You can use either Java or Python to create the custom logic. You must define the following properties in the File Set definition:

Name: Specify a unique name that is used to identify the File Set.
Type: Specify whether the File Set is used to split or annotate data.
File type: Specify whether the file is Java or script.
File name: Specify the name of the file that contains the custom logic that you defined. For example, if you use Java, this file is a Java Archive (JAR) file.
Class name: If you use Java, specify the name of the main Java class name.

Note:

Data sources, such as data source definitions, are not defined as part of a custom Insight Pack because data sources require specific information, such as host name, log path, and service topology information that is dependent on the server and environment. This information varies depending on where IBM Operations Analytics - Log Analysis is installed. As a result, when you define a custom Insight Pack, you only need to define data types such as Collections, Source Types, and Rule and File Sets.

After you install your custom Insight Pack, you must define the required data sources. For more information about how to create data sources, see the Administering IBM Operations Analytics - Log Analysis section.