Handling large input messages

Although a message flow developer cannot usually change the size of the input data or the model that leads to large memory usage, they can influence how much of a message is actively storied in memory at the same time.

In previous examples, fully parsed message trees were discussed. A standard full parse of a message tree parses a bitstream from start to finish, and fully inflates the message tree in memory. When this process produces a large message tree, large message handling techniques can be implemented to reduce memory usage.

The large message handling techniques presented here work on the principle that any large message tree is large because it has many repeating records. As such, the logic that is implemented by a message flow deals with each record one at a time and undertakes the necessary processing on that record. Because only one record is ever viewed at a time, only that one record needs to be held in memory. This process is achieved by the following sequence:

1. Using partial parsing to parse the bitstream in a message tree.
2. Parse the first repeating record with a reference variable (see References & navigating the message tree for more information).
3. Move to the next repeating record and delete the parent field of the previous record.

If an input message was large because it contained an embedded video or audio file for example, then this message is a single field, not many fields. Therefore, you cannot use these methods to reduce memory usage. As the whole structure must be populated in the message tree, there is typically less capacity to optimize processing. However, you can still take advantage of the Configuration Recommendations to Reduce Memory Usage Best Practice PDF document, or Configuration Recommendations to Reduce Memory Usage in the main product documentation.

By deleting the parent field of the record after it was processed, the elements that were being used for that record are made available to the parser to use again. This behavior means that when the next repeating record is fully parsed, it reuses the same underlying message tree field objects instead of creating new ones. Using this technique means that only the memory for one record is ever used, irrespective of how many repeating records are in the input message.

In summary, large message handling techniques can reduce how many message tree fields are in memory at the same time. Deleting fields does not apply only to repeating fields, it can also apply to any message tree field. Because the DELETE FIELD command has an overhead when the field is removed from the tree hierarchy, a message flow developer might not want to do this for every field that is ever referenced.