IBM Integration Bus, Version 9.0.0.5 Operating Systems: AIX, HP-Itanium, Linux, Solaris, Windows, z/OS

See information about the latest product version

Parser creation and memory usage

Partial Parsing, sometimes referred to as on-demand parsing, is the most efficient way to parse in IBM® Integration Bus.

For more information relating specifically to partial parsing, see Partial parsing in the Parsing strategies section. The rest of this section discusses parsers and their memory usage in general.

Messages are parsed from the start of the message, and proceeds as far along the message as required to access the element that is being referred to in the flow though the processing logic (ESQL, Java™, XPath, Graphical Data Mapper map, and so on). Depending on the field that is being accessed, it might not be necessary to parse the whole message. Only the portion of the message that is parsed is populated in the message tree. The rest is held as an unprocessed bitstream. The remainder might be parsed later in the flow if there is logic that requires it, or it might never be parsed if it is not required as part of the message flow processing.

When a message flow receives message data in one of its input nodes, parsers are created to handle that incoming data. Multiple parsers are created and chained to own each of the transport headers and then a body parser, such as XMLNSC, DFDL, or JSON, is created to own the message body. IBM Integration Bus also creates a Properties parser to represent the properties of the message tree, and a Root parser is created to own all these parser instances.

For example, when a message arrives on an MQInput node, you might have an incoming WebSphere® MQ message with the following message structure:

Root
 - Properties
   - ...
 - MQMD
   - ...
 - MQRFH2
   - ...
 - MQCIH
   - ...
 - XMLNSC
   - ...

This message tree has 6 parsers that represent it. For each main folder that is created from Root by a built-in node, a parser is created to own that folder.

The following example shows the message tree in full:

['MQROOT' : 0x642f630]
  Properties = ( ['MQPROPERTYPARSER' : 0x6366008]
  MessageSet             = '' (CHARACTER)
  MessageType            = '' (CHARACTER)
  MessageFormat          = '' (CHARACTER)
  Encoding               = 546 (INTEGER)
  CodedCharSetId         = 437 (INTEGER)
  Transactional          = TRUE (BOOLEAN)
  Persistence            = FALSE (BOOLEAN)
  CreationTime           = GMTTIMESTAMP '2015-01-14 13:14:12.350' (GMTTIMESTAMP)
  ExpirationTime         = -1 (INTEGER)
  Priority               = 0 (INTEGER)
  ReplyIdentifier        = X'000000000000000000000000000000000000000000000000' (BLOB)
  ReplyProtocol          = 'MQ' (CHARACTER)
  Topic                  = NULL
  ContentType            = '' (CHARACTER)
  IdentitySourceType     = '' (CHARACTER)
  IdentitySourceToken    = '' (CHARACTER)
  IdentitySourcePassword = '' (CHARACTER)
  IdentitySourceIssuedBy = '' (CHARACTER)
  IdentityMappedType     = '' (CHARACTER)
  IdentityMappedToken    = '' (CHARACTER)
  IdentityMappedPassword = '' (CHARACTER)
  IdentityMappedIssuedBy = '' (CHARACTER)
  )
  MQMD       = ( ['MQHMD' : 0x63433c8]
  SourceQueue      = 'IN' (CHARACTER)
  Transactional    = TRUE (BOOLEAN)
  Encoding         = 546 (INTEGER)
  CodedCharSetId   = 437 (INTEGER)
  Format           = '        ' (CHARACTER)
  Version          = 2 (INTEGER)
  Report           = 0 (INTEGER)
  MsgType          = 8 (INTEGER)
  Expiry           = -1 (INTEGER)
  Feedback         = 0 (INTEGER)
  Priority         = 0 (INTEGER)
  Persistence      = 0 (INTEGER)
  MsgId            = X'414d512042524b36312020202020202006e06d4920001a04' (BLOB)
  CorrelId         = X'000000000000000000000000000000000000000000000000' (BLOB)
  BackoutCount     = 0 (INTEGER)
  ReplyToQ         = '                                                ' (CHARACTER)
  ReplyToQMgr      = 'V9IN                                           ' (CHARACTER)
  UserIdentifier   = 'vgrover    ' (CHARACTER)
  AccountingToken  = X'1601051500000060022a3f33997b361af17567f401000000000000000000000b' (BLOB)
  ApplIdentityData = '                                ' (CHARACTER)
  PutApplType      = 11 (INTEGER)
  PutApplName      = 'rfhutil.exe' (CHARACTER)
  PutDate          = DATE '2015-01-14' (DATE)
  PutTime          = GMTTIME '13:14:12.350' (GMTTIME)
  ApplOriginData   = '    ' (CHARACTER)
  GroupId          = X'000000000000000000000000000000000000000000000000' (BLOB)
  MsgSeqNumber     = 1 (INTEGER)
  Offset           = 0 (INTEGER)
  MsgFlags         = 0 (INTEGER)
  OriginalLength   = -1 (INTEGER)
  )  
MQRFH2 {
  MQCHAR4  StrucId;    
  MQLONG   Version;        
  MQLONG   StrucLength;    
  MQLONG   Encoding;       
  MQLONG   CodedCharSetId; 
  MQCHAR8  Format;         
  MQLONG   Flags;          
  MQLONG   NameValueCCSID;
  } MQRFH2; 
 )
  MQCIH      = (
  Version            = 2
  Format             = '        '
  Encoding           = 2
  CodedCharSetId     = 437
  Flags              = 0
  ReturnCode         = 0
  CompCode           = 0
  Reason             = 0
  UOWControl         = 273
  GetWaitInterval    = -2
  LinkType           = 1
  OutputDataLength   = -1
  FacilityKeepTime   = 0
  ADSDescriptor      = 0
  ConversationalTask = 0
  TaskEndStatus      = 0
  Facility           = X'0000000000000000'
  Function           = '    '
  AbendCode          = '    '
  Authenticator      = '        '
  Reserved1          = '        '
  ReplyToFormat      = '        '
  RemoteSysId        = '    '
  RemoteTransId      = '    '
  TransactionId      = '    '
  FacilityLike       = '    '
  AttentionId        = '    '
  StartCode          = '    '
  CancelCode         = '    '
  NextTransactionId  = '    '
  Reserved2          = '        '
  Reserved3          = '        '
  CursorPosition     = 0
  ErrorOffset        = 0
  InputItem          = 0
  Reserved4          = 0
  )
  
  XMLNSC     = ( ['xmlnsc' : 0x63669b8]
  root = (
  Folders  = (
  Folder1 = (
  SubField = '1' (CHARACTER)
        )
        Folder2 = (
        SubField = '1' (CHARACTER)
        )
      )
      Fields   = (
      Field1 = 'field1Value' (CHARACTER)
      Field2 = 'field1Value' (CHARACTER)
      Field3 = 'field1Value' (CHARACTER)
      )
      Elements = (
      Field1  = 'field1Value' (CHARACTER)
      Folder1 = (
      SubField = '1' (CHARACTER)
        )
      Field2  = 'field1Value' (CHARACTER)
      Field3  = 'field1Value' (CHARACTER)
      Folder2 = (
      SubField = '1' (CHARACTER)
        )
     )
   )
)

If this message tree goes through a transformation node, then a new output message tree is created that might have none, some or all of the contents of the input message tree. This output tree is a separate message tree from the input message tree. As such, a new set of parsers are created to own the parser folders in the output tree. That is, there is not just one parser per domain, multiple instances can be created based on the message flow logic

So in the previous example, if this tree were copied with ESQL such as SET OutputRoot = InputRoot; then you would now have 12 parsers that were created by this message flow. It is clear to see that the more transformation nodes that are used, the more parsers are created to own the new output message trees. Therefore, when you consider copying message trees, the cost of the extra parsers must be considered.

The request nodes, such as the SOAPRequest nodes, are similar to a transformation node, in that a different message tree is propagated to the one that is received on the input node. The input message tree is used to form a request message for which a response to is received, usually from a remote server. The response message is then parsed using the parser parameters that are specified on the request node. This sequence means that the propagated response message creates a new set of parsers such as Root, Properties, and Header parsers, and a body parser. A message flow developer does not often have any choice on which request nodes are used in a message flow, and as such cannot make any improvements in this area.

The transformation nodes also allow a message flow developer to create parser instances manually. This creation can be done explicitly using method/function call:

In ESQL the CREATE with the DOMAIN clause creates a parser of the named domain.
In Java the MbElement create*UsingParser methods create a parser of the named parser name.
In .Net the NbElement create*UsingParser methods create a parser of the named parser name.
In the C Plugin interface the cniCreate*UsingParser methods create a parser of the named parser name.

The creation can also be done implicitly when constructing MbMessage, NbMessage, or CciMessage objects.

Although built-in transformation nodes construct one set of Output trees only, the Java, C, and .Net APIs allow many sets of output message objects to be created. Therefore it is important to understand the scope of these objects and associated parsers.

Consider a scenario where multiple bitstream portions are retrieved from an external source and need to be mapped into an output message as shown in the following ESQL example:

-- Bitstream portions are in Environment.Variables.MsgData[]
DECLARE outRef REFERENCE TO OutputRoot;
CREATE LASTCHILD OF outRef AS outRef DOMAIN('XMLNSC') NAME 'XMLNSC';
CREATE LASTCHILD OF outRef AS outRef NAME 'TestCase';
DECLARE envRef REFERENCE TO Environment.Variables.MsgData;
DECLARE parseOptions INTEGER BITOR(RootBitStream, ValidateNone); 
WHILE LASTMOVE(envRef) DO
  CREATE LASTCHILD OF Environment DOMAIN('DFDL') PARSE(envRef OPTIONS parseOptions TYPE '{}:CSV_DFDL');
  CREATE LASTCHILD OF outRef NAME 'Record';
  SET outRef = Environment.DFDL;
  MOVE envRef NEXTSIBLING NAME 'MsgData';
END WHILE;

In this example, the CREATE with the DOMAIN clause creates a DFDL parser in each iteration of the WHILE loop. Each of these DFDL parsers is a new DFDL parser to the one on the previous iteration. Therefore, if this loop iterates 2000 times, then 2000 DFDL parsers are created. More parsers means more memory usage, so resources are rapidly consumed with this approach. The following sections describe ways to mitigate the memory problems that are associated with this kind of scenario.

bj60035_.htm |

Last updated: 2016-08-12 11:20:23