Editing the index configuration

Edit the basic metadata and add the indexing fields that comprise the index configuration for a project.

Before you begin

If you want to use an AQL module to define an index configuration, import the module before you start editing the index configuration.

About this task

When you create an Insight Pack project in Eclipse, the index configuration file (metadata\indexconfig.json) and the index configuration resource file (indexconfig_spec.ucdk) are automatically created.

Use the Index Configuration Editor to edit the index configuration resource file. You can also create new index configuration instances and edit them in the Index Configuration Editor.
Note: If you manually edit the metadata\indexconfig.json for a project that you have opened in the Log Analysis Insight Pack Tooling, any changes you make are not displayed and are overwritten by changes made within the Tooling.

Procedure

To edit the index configuration:

  1. In the Eclipse Navigator pane, double-click the indexconfig_spec.ucdk file to start the Index Configuration Editor.

    Alternatively, right-click on the file and choose Open from the context menu.

  2. In the Index Configuration Editor Overview tab, you can view the basic metadata for the default project Index Configuration.

    The metadata fields are represented as strings in the index configuration file.

    The following metadata fields are available:
    Table 1. Index configuration data
    Metadata JSON string Description
    Name name Required. Specifies the name of the index configuration.

    The default is <projectname>_indexconfig.

    Version version Required. Specifies the index configuration version and must be four digits.

    For example, 1.1.0.0.

    Description description Required. Contains a description of the index configuration.

    The default description is Index configuration - <projectname>.

  3. Optional: If you want to create another index configuration instance, click Add. To use an existing Index configuration as a basis for a new Index configuration, select a Index configuration and click Copy. The copied Index configuration instance is displayed in the Overview tab and is named with the prefix CopyOf. Edit the name in the Attributes field. Make any changes you require to the Index configuration fields in the Field configuration tab before you proceed. You can also delete an instance by highlighting the instance name and clicking Remove.

    In the NewIndexConfig dialog, complete the basic metadata index configuration for the new instance and click Finish.

  4. The Index Configuration Editor is also used to configure the fields within the Index Configuration instance.
    The following fields are required for all index configuration instances; they are created automatically, with default values, when a new index configuration instance is created.
    • timestamp
    • logRecord

    To add one or more indexing fields to the index configuration:

    1. Open the Field Configuration tab. If you have more than one Index configuration instance, choose the instance you want to edit from the Index configuration instances list.
    2. Click Add to add an indexing field to the index configuration. To use an existing indexing field as a basis for a new indexing field, select an indexing field and click Copy. The copied indexing field is named with the prefix CopyOf. Edit the name in the Attributes field. Make any additional required changes to the attribute fields before you proceed.
    3. In the New Field Configuration dialog, select the field attributes that you require.

      When you select a check box to choose an attribute, the attribute is assigned the value true in the index configuration file. Unselected attributes are assigned the value false.

      The attributes are represented as strings in the index configuration file.

      The following attributes are available:
      Table 2. Index configuration attributes
      Attribute JSON string Description
      Name User-defined. Required. Specifies the field name.
      Data type dataType Specifies the field data type. You can choose one of the following data types:
      • TEXT
      • LONG
      • DOUBLE
      • DATE

      The default is TEXT.

      Retrievable retrievable Optional. Determines whether the contents of the field are stored in the index for later retrieval.

      When you select this attribute, the contents of the field are not directly searchable but they are returned within query results that match any other searchable field in a log record.

      The default is false.

      Retrieve by default retrieveByDefault Optional. Determines whether the contents of the field are returned by default as part of any search response.

      This attribute is only available when you select the Retrievable attribute.

      If you do not select this attribute, the contents of the field are still returned in search responses when explicitly requested.

      The default is false.

      Sortable sortable Optional. Determines whether the contents of the field can be sorted and included in range queries.

      The default is false.

      Filterable filterable Optional. Determines whether facet counting and filtering is enabled for the contents of the field.

      The default is false.

      Searchable searchable Optional. Determines whether the contents of the field are returned by search queries.

      The default is true.

    4. When you are finished selecting attributes, click Next.
    5. Click Add to add source details for the field. You can also Edit or Remove source details.
    6. In the New Field Source dialog, enter the source path.
      Note: The paths must be prefixed with metadata, content. or annotations. For example, metadata.logsource or content.text

      You create the field path in the following format:

      annotations.<modulename>_<viewname>.<viewfieldname>

      where <modulename> is the AQL module name and <viewname> is the name of an output view in the AQL module. <viewfieldname> is a field in the output view. This field must be a text type.

      For example, consider the following AQL sample:
      module Unityannotator;
      
      create view UnityLog as
      extract regex /.*\d\d\s\[(.*)\]\s([A-Z]*)\s*-\s*([A-Za-z]*)\s*:\s*(.*)/
      on D.text return
      group 1 as ThreadNo and
      group 2 as Severity and
      group 3 as msgClass and
      group 4 as Message
      
      from Document D;
      
      export view UnityLog;
      You create the following source paths for this AQL sample:
      annotations.Unityannotator_UnityLog.ThreadNo.text
      annotations.Unityannotator_UnityLog.Severity.text
      annotations.Unityannotator_UnityLog.msgClass.text
      annotations.Unityannotator_UnityLog.Message.text
      In the above source paths, the AQL module is Unityannotator. The view name is UnityLog. The view field names are ThreadNo.text, Severity.text, msgClass.text, and Message.text.
    7. If you select Date as the data type, you must select a date format. The Date format field is only displayed for the Date data type.

      Any date format consistent with the Java Simple Date Format is valid.

      Examples of supported date formats include the following, but are not limited to these example formats:
      WebSphere® Application Server logs date format
      [MM/dd/YY HH:MM:ss:SS z]
      Generic annotation log date format
      [YYYY/MM/dd HH:mm:ss,z]
      DB2 date format
      YYYY-MM-dd-HH-mm.ss.SSS Z

      For more information about date/time masks and format specifiers, see:

      http://www-01.ibm.com/support/knowledgecenter/SSMQ79_9.0.1/com.ibm.egl.lr.doc/topics/regl_core_date_format.html

    8. Click Finish.
    9. Choose a source combination. The available options are ALL and FIRST.

      The Combine field only becomes available when you have already created two or more sources.

    10. In the New Field Configuration dialog, click Finish.

      The field attributes are displayed in an Attributes pane in the Field Configuration tab. After you create a field, you can modify the attributes in the Attributes pane.

  5. Create more indexing fields, if required.
  6. To remove a field, select it from the list of fields in the Fields pane and click Remove.
  7. Save the index configuration.

Results

The index configuration file is updated with the new metadata and field details.