IBM Support

Overview of ISIM request retry process

Question & Answer


Question

How does ISIM handle requests submitted during reconciliations, and requests submitted for inactive adapters?

Answer

Please see the note at the end for a new option in ISIM 6.0/7.0 to handle blocked services.

There are three main reasons why requests might build up in the Pending Requests queue: problems with the messaging engine, problems with the database, and requests delayed by ISIM for later processing. This technote deals solely with requests intentionally delayed by ISIM.

Delayed requests are controlled by two tables in the ISIM database: RESOURCE_PROVIDERS and REMOTE_SERVICES_REQUESTS. Each Service in ISIM has an entry in the RESOURCE_PROVIDERS table. The relevant columns for this issue are: RECON_STATUS, LOCK_SERVICE, and RESOURCE_STATUS. When a request is delayed, a reference is stored in REMOTE_SERVICES_REQUESTS where the relevant columns are: PROVIDER_ID, TYPE, and REQUEST_TIME.

There are four TYPEs of requests that can be stored in REMOTE_SERVICES_REQUESTS. They are:

0 - generic requests (not used)

1.- asynchronous requests (e.g. Universal Provisioning Adapter messages)

2 - intra-reconciliation requests (submitted during a recon)

3 - service deferred requests (submitted while an adapter is down)

The first two types are rarely seen in practice, so only the latter two will be dealt with here.

When a reconciliation is run:

- If the service is not locked, the request will be sent, and both ISIM and the adapter will update the recon results as needed.

- If the service is locked (LOCK_SERVICE=1), then any requests for this service will be stored in REMOTE_SERVICES_REQUESTS with TYPE=2.

Once the reconciliation finishes successfully, ISIM will check to see whether there are any entries for the PROVIDER_ID of this service and where TYPE=2. If any are found, they will be immediately executed.

However, if the reconciliation crashes or otherwise ends in failure, none of the TYPE=2 requests will be processed. There are two ways to correct this situation:

1. ISIM will process the requests after the next successful reconciliation of this service. Once the original reconciliation problem has been corrected, a full or filtered reconciliation can be run. If that completes successfully, the built up requests will be processed.

2. If these requests are no longer needed, they can be manually deleted. This approach should only be undertaken with the assistance of IBM Support. Before deleting the entries in REMOTE_SERVICES_REQUESTS where PROVIDER_ID=<this service> and TYPE=2, the corresponding entry in the PROCESS and/or ACTIVITY tables must be updated to a completed status based on the REQUEST_ID.

The following entries in $ISIM_HOME/data/enRole.properties control how requests for inactive adapters are processed:


###########################################################
## Deferring requests for failed Remote Resources
###########################################################
# Whether to defer requests to failed resources and wait for
# resource to restart before sending them.
# If false, requests will terminate immediately if resource has failed.
com.ibm.itim.remoteservices.ResourceProperties.DEFER_FAILED_RESOURCE=true

# scheduled interval to check failed resources to see whether
# they have been restarted (seconds).
# Minimum = 120, Maximum = 3600, Default = 600
remoteservices.remotepending.interval=600

# maximum time a request to a resource can be outstanding,
# including time in pending state (for Asynchronous requests,
# or deferred requests due to service failure or backlog).
# Use -1 for unlimited (in this case workflow activity timeouts
# will define actual maximum duration - this is set in the workflow
# process definitions).
# Default = -1 (unlimited), minimum = 60 + value of remoteservices.remotepending.interval
com.ibm.itim.remoteservices.ResourceProperties.MAX_REQUEST_TIME=-1

# The amount of time in minutes that pending requests generated from the
# restart of a failed service are given to complete. At the conclusion
# of this time period, the server will retry these requests.
remoteservices.remotepending.restart.retry=1440


This processing will only take place if DEFER_FAILED_RESOURCE is set to TRUE. If it is set to FALSE, the request will fail if the adapter cannot be reached. With this feature enabled, if ITIM is unable to connect to an adapter, the request will be stored in REMOTE_SERVICES_REQUESTS to await the return of the adapter. This ONLY applies to connection errors. If the adapter accepts the connection, but something prevents it from receiving or processing the request, that will NOT trigger this behavior.

The first time ISIM cannot connect to an adapter, it will set the RESOURCE_STATUS=1 for this service in the RESOURCE_PROVIDERS table, and the request involved will be placed in REMOTE_SERVICES_REQUESTS with TYPE=3. For every request, ISIM checks the RESOURCE_STATUS column before contacting the adapter. If it is set to "1" for this service, then the new request will be immediately placed in the REMOTE_SERVICES_REQUESTS table, and it will show in the Pending Requests queue.

It is important to note that Reconciliations and Test connections do NOT check the RESOURCE_STATUS column, so they will execute even if other requests are piling up. A reconciliation will check the RECON_STATUS column of RESOURCE_PROVIDERS, and if that is set to "1", then it will fail with a message that another reconciliation is currently running. Since ISIM trusts the values in this table, it is possible to receive this message when a reconciliation is not running if a prior reconciliation failed in such a way that the RECON_STATUS was incorrectly left as "1". A related link has been provided on how to deal with that situation. The Test button on the Service Form also ignores the database. It simply opens a socket to the URL specified, and reports whether it connected successfully or not. A successful Test indicates the adapter has been restored, but it will not update the RESOURCE_STATUS in the database, nor will it trigger the execution of any built up requests.

Based on the "remotepending.interval" setting, every 10 minutes by default, ISIM will query the RESOURCE_PROVIDERS table for any rows where RESOURCE_STATUS=1. For each one it finds, it will set the RESOURCE_STATUS=2 (testing). It will then look in REMOTE_SERVICES_REQUESTS for any rows where TYPE=3 for this PROVIDER_ID, sorted by REQUEST_TIME. The first request found will be sent to the adapter for processing.

If the request completes successfully, the RESOURCE_STATUS will be set to 0, and the rest of the requests where TYPE=3 for this PROVIDER_ID will be executed.

If the request does not complete successfully, for ANY reason, the RESOURCE_STATUS will be set back to 1, and the process will repeat in another 10 minutes. Typically the failure is due to the fact the adapter is still down, so restarting it can usually fix this. However, it is also possible there is something wrong with the first request - e.g. it's a restore of an active account, or adding an account that already exists (perhaps because it was added manually while the adapter was down). In that case, this loop will continue until the administrator corrects it. IBM Support should be contacted to assist with the cleanup, but the general steps involve updating the PROCESS, PROCESSLOG, PROCESSDATA, and ACTIVITY tables where PROCESS_ID=REQUEST_ID, then deleting that entry from REMOTE_SERVICES_REQUESTS. As long as the next request in line is able to complete successfully, the problem will be resolved. If not, all of the problem requests must be deleted until one can complete successfully.

Please note that setting RESOURCE_STATUS=0 in RESOURCE_PROVIDERS will allow new requests to be sent to the adapter, but this will NOT allow previously submitted requests to complete. If you have set RESOURCE_STATUS=0 as a temporary workaround, you must set RESOURCE_STATUS back to "1", then wait up to 10 minutes for ISIM to look for failed services and go try the first request in REMOTE_SERVICES_REQUESTS. Entries in REMOTE_SERVICES_REQUESTS where TYPE=3 will ONLY be executed as part of the ISIM request retry process described above.

Note for ISIM 6.x,7.x:
Starting in ISIM 6.0, a new feature was added to "Retry Blocked Requests". By default, this option will only be seen if the user is the Service Owner, or an Administrator. It is controlled by both a View and an ACI, so if you do not see this option available for a regular service, then those would be the first things to check. This option does not apply to the ISIM service, Manual Services, or Hosted Services.

If requests are pending for a service that's currently marked as down, then once the adapter has been restored to a working state, you can navigate to Manage Services and select "Retry Blocked Requests" on the target service rather than waiting for the automatic retry process to process it. When you select this option, ISIM will first run a Test Connection to the adapter. If that succeeds, it will then attempt to process the first pending requests. If this also succeeds, then ISIM will submit the rest of the pending requests for processing, and everything will go back to normal. However, if the request fails for any reason, similar to the automatic process, then the request will be put back on pending queue and the service will be left marked in a down state. In that case, the reason for the failure must be investigated, as no other requests for this service will process until this first request is cleared, or completes successfully.

[{"Product":{"code":"SSRMWJ","label":"IBM Security Identity Manager"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Not Applicable","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"6.0;5.1;7.0","Edition":"Enterprise","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Product Synonym

enrole itim tim isim sim

Document Information

Modified date:
16 June 2018

UID

swg21380983