Why use SCM
Emergency storage and improved performance are two use cases for using SCM with IBM® MQ for z/OS®.
Emergency storage
SMDS and message offloading can be used in conjunction with SCM to reduce the likelihood of an MQRC_STORAGE_MEDIUM_FULL reason code being returned to an IBM MQ application during an extended outage.
Overview
A single shared queue is configured on an application structure. The putting application puts messages onto the shared queue; the getting application gets messages from the shared queue.
During normal running, the queue depth is expected to be close to zero, but a business requirement indicates that the system must be able to tolerate a two-hour outage of the getting application. This means that the shared queue must be able to contain two hours of messages from the putting application.
Currently, this process is achieved by using the default offload rules, and SMDS, so that the size of the structure is minimized, while reducing the performance cost that is associated with offloading.
The rate of messages being sent to the shared queue is expected to double in the short to medium term. Although the requirement that the system be able to tolerate a two-hour outage still exists, not enough real storage is available in the CF to double the size of the structure.
Because the CF, which contains the application structure, resides on a zEC12 machine, the possibility exists to associate sufficient SCM with the structure to store enough messages so that a two-hour outage can be tolerated.
- Initially, the system is in its steady state. Both the putting and getting application are running normally and the queue depth is close to, or at, zero. The result is that the application structure is largely empty.
- At a certain time, the getting application suffers an unexpected failure and stops. The putting application continues to put messages to the queue and the application structure starts to fill up.
- After the structure reaches 70% full, the conditions of the first offload rule are met and all
messages with a size greater than or equal to 32 KB are offloaded to SMDS.
See SMDS and offload rules for an overview of the offload rules.
- As messages continue to be put to the shared queue, the structure continues to fill (either
because of the message data being stored in the structure, or as a result of the pointers to the
offloaded messages being stored in the structure).
When the structure reaches 80% full, the second offload rule starts to apply and messages that are 4 KB or greater are offloaded to SMDS.
- When the structure is past 90% full, all messages are offloaded to SMDS and only the message
pointers are being placed in the structure.
About this time, the pre-staging algorithm starts to run, and begins moving data from the structure into SCM. Assuming all messages on the queue are the same priority, the newest messages are pre-staged.
Because all messages are now being offloaded to SMDS, the data being moved into SCM is not actual message data, but instead the pointers to the messages on SMDS.
As a result, the number of messages that can be stored on the combination of the structure, and the SCM and SMDS associated with the structure, is very large.
Performance: During this stage of the outage, the putting application can suffer a degree of performance degradation because of having to write to SMDS. In this case, the use of SCM should not be a limiting factor on the putting application in terms of performance. SCM provides extra space to prevent the structure filling up. - Eventually the getting application is available again and the outage is over.
However SCM is still being used by the structure. The getting application starts reading messages off the queue, getting the oldest, highest priority messages first.
Because these messages were written before the structure started to fill up, they come out entirely from the real storage portion of the structure.
- As the structure starts to empty, it goes below the threshold at which pre-staging is active, and so pre-staging stops.
- The structure usage reduces below the point at which the offload rules take effect, so messages
are no longer offloaded to SMDS unless they are more than 63 KB.
At about this time, the pre-fetch algorithm starts moving data from SCM in to the structure. Because the getting application gets messages from the queue in the order expected by the SCM algorithms, messages are brought in before the getting application needs them.
The result is that the getting application never needs to wait for messages to be brought in synchronously from SCM.
- As the getting application continues to move down the queue, it starts retrieving messages that were offloaded to SMDS.
- Finally, the system is in a steady state again. No messages are stored in SCM or SMDS, and the queue depth is close to zero.
Improved performance
This scenario describes using SCM to increase the number of messages that can be stored on a shared queue without incurring the performance cost of using SMDS.
Description
For this scenario, a putting and getting application communicate through a shared queue which is stored in application structure.
The putting application tends to run in bursts, when it puts a large number of messages in a short amount of time. Then, in an extended period of time, it produces no messages at all.
The getting application sequentially processes each message, and performs complex processing on each one. As a result, most of the time the queue depth is zero, except for when the putting application starts to run, where the queue depth starts increasing as messages are being put faster than they are being got.
The queue depth increases until the putting application stops, and the getting application has enough time to process all the messages on the queue.
- In this scenario, the key factor is performance. The messages being sent to the queue are always less than 63 KB and so never need to be offloaded to SMDS.
- The application structure has been sized so that it is large enough to contain all of the messages that will be placed on it by the putting application in a single “burst.”
- The offload rules must all be disabled so that, even when the structure starts to fill, the messages are not offloaded to SMDS. This is because the performance costs that are associated with writing messages to, and reading messages from, SMDS are deemed unacceptable.
Over time, the number of messages that the putting application send in a burst must increase by several orders of magnitude. Because the getting application must process each message sequentially, the number of messages on the queue increases to the point where the structure fills up.
At this point, the putting application receives a reason code (MQRC_STORAGE_MEDIUM_FULL) when putting a message, and the put operation fails. The putting application can only briefly tolerate periods when it is unable to put messages to the queue. If the period is too long, the application ends.
- Increase the size of the application structure.
- Add offload rules to the application structure so that messages are offloaded to SMDS as the queue starts to fill up.
- Associate SCM with the structure.
The first solution is quick to implement, but not enough real storage is available on the CF.
The second solution might also be quick to implement, but the performance impact of offloading to SMDS is considered too significant to use this option.
The third solution, associating SCM with the structure, provides an acceptable balance of cost and performance.
Associating SCM with a structure results in a higher use of real storage in the CF because of the augmented storage that get operations used. However, the actual amount of real storage will be less than the amount used in the first option.
Another consideration is the cost of SCM. However this cost is much cheaper than real storage. These factors combine to make the third option cheaper than the first option.
Although the third option, potentially, might not perform as well as the first option, the pre-fetch and pre-staging algorithms used by the CF can combine to make the differences in performance acceptable, or in some cases negligible.
Certainly the performance can be much better than using SMDS to offload messages.
- Initially, the getting application is active and waiting for messages to be delivered to the shared queue. The putting application is not active and the shared queue is empty.
- At a certain time, the putting application becomes active, and starts putting a large number of
messages to the shared queue. The getting application starts getting the messages, but the queue
depth rapidly starts to increase because the getting application is slower than the putting
application.
As a result, the application structure starts to fill up.
- As the time increases, the putting application is still active. The application structure fills
up to approximately 90%.
This is when the SCM pre-staging algorithm starts to move messages from the structure into SCM, freeing space in the structure.
Because the getting application gets the oldest, highest priority messages from the queue first, it is always getting messages from the structure and does not need to wait for messages to be brought synchronously from SCM in to the structure.
- The putting application is still active and putting messages to the shared queue. However, the application never receives an MQRC_STORAGE_MEDIUM_FULL reason code, because enough space exists in SCM to store all the messages that do not fit in the structure.
- Eventually, the putting application stops because it has no more messages to put.
The pre-staging algorithm stops because the structure falls below 90% in use, and the getting application continues processing the messages in the queue.
- As the getting application starts to free space in the structure, the pre-fetch algorithm starts
to bring messages back from SCM in to the structure.
Because the getting application processes messages in the order expected by the pre-fetch algorithm, the getting application never becomes blocked waiting for message data to be brought synchronously from SCM in to the structure.
- Finally, the getting application processes all the messages on the shared queue, and waits until the next message is available. The structure and SCM are empty of messages.