z/OS log data is missing in Log Analysis search results

Determine whether the z/OS Log Forwarder logged any error or warning messages to its STDERR data set. Most error and warning messages are also written to the console, but any available exception and stack trace information is written only to STDERR.

• Using the message ID, look up the message in the message reference documentation. Review the explanation, and take appropriate action.

If no z/OS Log Forwarder error or warning messages are logged, review the UNITY_HOME/logs/GenericReceiver.log file on the IBM SmartCloud Analytics - Log Analysis server.

• Look for messages with severity ERROR or WARN. For more information about any error or warning messages that you find, see the IBM SmartCloud Analytics - Log Analysis documentation.

The UNITY_HOME/logs/GenericReceiver.log file also specifies the number of log records that are processed in each batch of log data, as shown in the following example:

03/06/14 15:58:25:660 EST [Default Executor-thread-4] INFO  - DataCollectorRestServlet : 
Batch of Size 9 processed and encountered 0 failures

In this example, nine log records were successfully ingested, and no log records failed to be ingested.

You can determine the data source name by looking for a data source ingestion record before the batch status record in the GenericReceiver.log. In the following example, the data source name is my_sysprint:

03/06/14 15:58:25:650 EST [Default Executor-thread-4] INFO  - UnityFlowController : 
Adding data source under ingestion, logsource = my_sysprint threadId = 96

Determine whether the status information in the GenericReceiver.log indicates one of the following three situations:

All batch status records for a data source have a size value of 0

Log data is being sent to IBM SmartCloud Analytics - Log Analysis, but one of the following issues is preventing any log records from being identified:

• The data source type that is specified for the data source that is configured in IBM SmartCloud Analytics - Log Analysis is incorrect.
  Examples:

  • The data source type is WASforzOS-SYSOUT, but the log data is being sent from a SYSPRINT data set. Therefore, the data source type must be specified as WASforzOS-SYSPRINT.
  • The data source type is WASforzOS-SYSPRINT, and the log data is being sent from a SYSPRINT data set, but the SYSPRINT data set contains log data in the distributed format. Therefore, the data source type must be specified as WASSystemOut.
• Although the data source type is correct, the log data does not contain any records in the format that is expected by the data source type.
  Examples:

  • The SYSPRINT data contains only WebSphere® Application Server internal trace records. It does not contain records that are produced by the Java™ RAS component. For SYSPRINT, only records that are produced by the Java RAS component are ingested.
  • The SYSPRINT data contains only unstructured data, such as data that is produced by System.out.println() calls from Java code. For SYSPRINT, only records that are produced by the Java RAS component are ingested.
  • The SYSOUT data contains Java garbage collector trace data. Java garbage collector data is not supported by the WASforzOS-SYSOUT data source type.

Batch status records are present for the data source, the batch size is nonzero, and the number of failures is zero

Log records are being ingested successfully. Determine whether you have one of the following situations:

• The search criteria is incorrect or not broad enough to include the expected log data.
  Example: If the search time filter is set from 2:00 PM until 2:05 PM, a record that was logged at 2:05:01 PM is not flagged.

  Verify the search filters. You might want to start with a broad search, and refine it as needed.

• The time zone information is incorrectly configured in the z/OS Log Forwarder. This issue can cause the time stamps of the ingested records to be 1 - 12 hours earlier or later than they should be.
  Verify the values of the following variables:

  • TZ environment variable in the z/OS Log Forwarder start procedure
  • timeZone values in the z/OS Log Forwarder configuration file
• If only the most recent log record is missing, the record might be held in buffer by IBM SmartCloud Analytics - Log Analysis. IBM SmartCloud Analytics - Log Analysis cannot confirm whether a log record is complete until the next log record is received for that data source. Therefore, for each data source, the last log record that is sent to IBM SmartCloud Analytics - Log Analysis is typically held (and not ingested) until the next log record is received.

No batch status records exist for the data source

Determine whether you have one of the following situations:

• The z/OS Log Forwarder is started, but no log records are generated for the data source. No log records are ingested because no log data exists to ingest.
• The user ID that is associated with the z/OS Log Forwarder start procedure does not have the appropriate access to files or directories.
  Example: For example, the user ID might not have read access to one of the following items:

  • A z/OS UNIX log file
  • The High Performance Extensible Logging (HPEL) log or trace directory.

    For more information about the HPEL log or trace directory, see WAS HPEL gatherer properties.

  Change the file permissions to give the user ID the appropriate access. For more information about the z/OS Log Forwarder user ID, see Creating the z/OS Log Forwarder start procedure.

• Because the log gatherer is incorrectly configured in the z/OS Log Forwarder configuration file, the z/OS Log Forwarder cannot find the log data.
  Example: For example, verify the following type of information:

  jobName for a z/OS Job Log gatherer

  If the jobName value is incorrect, the z/OS Log Forwarder cannot find any jobs.

  For more information, see z/OS Job Log gatherer properties.

  ddName for a z/OS Job Log gatherer

  If the ddName value is incorrect, the z/OS Log Forwarder cannot find any data sets.

  For more information, see z/OS Job Log gatherer properties.

  logPath for a z/OS UNIX Log File gatherer

  If the logPath value is incorrect, the z/OS Log Forwarder cannot find the log file.

  For more information, see z/OS UNIX Log File gatherer properties.

  Remember: The logPath value must be an absolute path.