IBM Support

DataPower off-device logging: a configuration example

Troubleshooting


Problem

This technote gives an example of how to enable off-device logging on an IBM® WebSphere® DataPower® appliance. This can be very helpful when a debug log level is needed to help isolate a problem or monitor behavior over a long period of time or can be used in production environment as DataPower only keeps a limited number of log files (the default is 3 files) in the file system in a rotational basis.

Cause

The DataPower device has a finite amount of space to hold larger than average log files or long term logging needs in production environment.

It should be noted that if used while debugging at load or in a capacity issue, log events may be dropped. There is a prioritization of events within the device and client traffic always comes first. Log events dropped can be confirmed or counted from the Status>Log Targets menu in the WebGUI.

Resolving The Problem

A few initial points that should be clear:
1- Off appliance logging has a few benefits, primarily when the appliance is not accessible or during issues and failures we can still retrieve log information.

2- The type of logging is just as important as the log event subscriptions. A lossy log target method will not help if a lot of events are dropped or lost.

3- In most environments three basic logs should be configured to capture off box or archive at least:

- Configuration events.

- A basic "all" "error" log target event subscription covering each domain.

- A latency log target is also an invaluable piece of information. Here is a technote on configuring a latency log.

The examples listed below will have their own strengths and weaknesses to be considered depending on the environment. Use the Logging Target status information in the appliance to see the statistics of the log target. The volume or events should be considered when selecting a protocol to use.

Syslog:

* The syslog protocol operates over UDP which has no guarantee on packet delivery - a fast fire and forget method.

To create the new log target, go into the default domain: Objects > Log Targets

Configure the log target with settings as follows from the WebGUI:

  1. Name the log target
  2. Select Target Type of syslog
  3. Fill in the Local Identifier with a descriptive string that may be used by a remote recipient to identify this specific log target
  4. Enter the Remote Host Address and the Remote IP Port as in the screen shot where x.x.x.x is the IP address of the remote syslog server that listens on port 514
  5. Take all other defaults


Under the Event Subscriptions tab, you can select all and debug as indicated here:


 

6. Generate log events in the DataPower by using some transactions, for example by saving the configuration from the WebGUI or running some test load into a domain.

 

Syslog-tcp is replacing Syslog-ng:
The older NG method was based on one TCP connection per log event and is being deprecated. The new tcp method reuses a persistent connection if the syslog server allows. The appliance will reuse TCP connections for improved performance and lower overhead on the appliance.

NFS:
All NFS limitations apply from file size, permission, and write performance/latency. This is a more common approach using an NFS static mount to capture a log target and can allow for slightly more reliable messages to be logged. The limiting factor again is the speed of the network and NFS server response time. NFS can be a heavy protocol for rapid or heavy appliance logging.

File type:

Using a file type log target with a backup method as seen here:



This will allow log events to quickly be written to a local file on the device's file system. Once the file reaches its set size a connection to the destination will upload the file from the device. This will upload with a unique time and date stamp on the uploaded file.

This is a useful method to capture sporadic problems. This is also useful for long running transactions that may span more than one file depending on device load.


HTTP service:
Finally a clever method that may be used in some senarios would be an HTTP service on the device.
Using an HTTP service configured in the following way,

Once the file type log is created in the logtemp:/// directory, a client browser or wget type client can easily retrieve the file from the device.

This is very useful when there is no local or accessible remote log storage location to the device, due to firewall or network restrictions.

This is also removing the limitation of the device dropping the log event should it not be able to write the log event to the network.

Each of the above methods are very useful and work better in some scenarios than others depending on the network, load, problem, and information needed. This is intended to be a guide to help you decide which method would be best for your scenario.

[{"Product":{"code":"SS9H2Y","label":"IBM DataPower Gateway"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Not Applicable","Platform":[{"code":"PF009","label":"Firmware"}],"Version":"4.0.2;5.0.0;6.0.0;6.0.1;7.0.0","Edition":"Edition Independent","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
24 August 2022

UID

swg21269136