Custom annotations and splitters

The general schema that is returned by the splitter contains the following attributes:

Log text

The text that is contained in the log record after it is split.

Timestamp

The timestamp, if there is one, that is associated with the log record.

Type

The type is a single character, A, B, or C, that indicates the type of this log record. The possible types are as follows:

• A: indicates a complete log record. The splitter logic determines that the associated record is complete. The record can be sent to the annotation and indexing processes. For example, in this example, the first record is a type A record and the second is of type B. This is because the second record indicates to the splitter that the first record is complete:

  [9/21/12 14:31:13:117 GMT+05:30] 0000003e InternalGener I 
    DSRA8203I: Database product name : D2/LINUXX8664
  [9/21/12 14:31:13:119 GMT+05:30] 0000003e InternalGener I 
    DSRA8204I: Database product version : SQL09070

• B: indicates that there is a partial log record at the end of the set. For example, the splitter detects the start of a new logical record but cannot determine if it is complete because the splitter cannot find the next logical record boundary that indicates the start of the next record. The splitter marks the record as type B to indicate to the IBM Operations Analytics - Log Analysis server that this record is a partial record and it must be buffered until more incoming records are received to allow it to complete the logical record. The IBM Operations Analytics - Log Analysis server sends all type A log records for annotation and indexing. It buffers type B records. The buffered type B records are then prefixed to the next batch of input that is sent to the splitter when it receives more input records. For example:

  [9/21/12 14:31:27:882 GMT+05:30] 00000051 servlet       
  E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E:
   Uncaught exception created in one of the 
  service methods of the servlet TradeAppServlet in application
   DayTrader2-EE5. Exception created : 
  javax.servlet.ServletException: TradeServletAction.doLogout
  (...)exception logging out user uid:1
  at org.apache.geronimo.samples.daytrader.web
  .TradeServletAction.doLogout(TradeServletAction.java:458)
  at org.apache.geronimo.samples.daytrader.web
  .TradeAppServlet.performTask(TradeAppServlet.java:169)
  at org.apache.geronimo.samples.daytrader
  .web.TradeAppServlet.doGet(TradeAppServlet.java:78)

• C: indicates that the text can be discarded. The IBM Operations Analytics - Log Analysis server discards this text. This type of record is not sent for annotation and indexing. It is not buffered. You must define the splitter so that it only marks text as type C if it is certain that it is not part of a log record that is not complete. For example, a partial log record is detected at the beginning of a batch of records. Then, a complete but unrelated logical log record is found. IBM Operations Analytics - Log Analysis can never complete the partial record that was detected first. The record must be marked as type C and discarded. For example:

  ************ Start Display Current Environment ************
  WebSphere Platform 7.0.0.0 [ND 7.0.0.0 r0835.03] running with process
   name cldftp48Node01Cell\cldftp48Node01\server1 and process id 28811
  Host Operating System is Linux, version 2.6.18-194.el5
  Java version = 1.6.0, Java Compiler = j9jit24, Java VM name = IBM J9 VM

For example, the annotation engine in IBM Operations Analytics generates the following JSON structure when it processes an AQL rule set against an incoming logical log record:

{ "annotations" : { "annotatorCommon_EventTypeOutput" :
  [ { "field_type" : 
     "EventTypeWS", 
           "span" : { "begin" : 57,
                "end" : 58,
                "text" : "E"
              },
            "text" : "E"
          } ],
      "annotatorCommon_LogTimestamp" :
         [ { "span" :
              { "begin" : 1,
                "end" : 32,
                "text" : "03/24/13 07:16:28:103 GMT+05:30"
              } } ],
      "annotatorCommon_MsgIdOutput" :
        [ { "field_type" :
            "MsgId",
            "span" :
              { "begin" : 59,
                "end" : 68,
                "text" : "DSRA1120E"
              },
            "text" : "DSRA1120E"
          } ],
      "annotatorCommon_ShortnameOutput" :
        [ { "field_type" : "ShortnameWS",
            "span" :
              { "begin" : 43,
                "end" : 56,
                "text" : "TraceResponse"
              },
            "text" : "TraceResponse"
          } ],
      "annotatorCommon_ThreadIDOutput" :
        [ { "field_type" : "ThreadIDWS",
            "span" :
              { "begin" : 34,
                "end" : 42,
                "text" : "00000010"
              },
            "text" : "00000010"
          } ],
      "annotatorCommon_msgText" :
         [ { "fullMsg" :
              { "begin" : 59,
                "end" : 167,
                "text" : "DSRA1120E: Application did not explicitly close 
all handles to this Connection. Connection cannot be pooled."
              },
                "span" : { "begin" : 70,
                "end" : 167,
                "text" : "Application did not explicitly close all handles 
to this Connection. Connection cannot be pooled."
              }
          } ]
    },
  "content" :
        { "span" : { "begin" : 1,
          "end" : 169,
          "text" :
           "[03/24/13 07:16:28:103 GMT+05:30] 00000010 TraceResponse 
E DSRA1120E: Application did not explicitly close all handles to this Connection. 
Connection cannot be pooled.\n"
        },
         "text" : "[03/24/13 07:16:28:103 GMT+05:30] 00000010 TraceResponse 
E DSRA1120E: Application did not explicitly close all handles to this Connection. 
Connection cannot be pooled.\n"
    },
  "metadata" : { "batchsize" : "506",
      "flush" : true,
      "hostname" : "mylogfilehost",
      "inputType" : "logs",
      "logpath" : "/data/unityadm/IBM/LogAnalysis/logsources/was/
SystemOut.log",
      "datasource" : "WAS system out",
      "regex_class" : "AllRecords",
      "timestamp" : "03/24/13 07:16:28:103 GMT+05:30",
      "type" : "A"
    }
}

In the example, there are three main sections or keys that are defined in the JSON data structure:

• Annotations: provide access to the annotation results that are created by the annotations engine when it processes an incoming log record according to AQL rules or custom logic.
• Content: provides access to the raw logical log record.
• Metadata: provides access to some of the metadata that describes the file that the log record was obtained from. For example, the host name or data source. In general, the metadata section contains any name/value pairs sent to the IBM Operations Analytics - Log Analysis server from a client along with the log data.

For example, to specify the text value for the annotated field MsgId from the previous example, use the following dot notation reference that references the actual value DSRA1120E:

annotations.annotatorCommon_MsgIdOutput.text

The following reference produces the same result:

annotations.annotatorCommon_MsgIdOutput.span.text

In a similar manner, you can use dot notation references to the content and metadata keys for the sourcepaths attribute value of each field to be indexed. For example:

content.text
metadata.hostname