Troubleshooting deadlocks

The following sections describe some of the most common deadlock scenarios and suggestions on how to avoid them.

com.ibm.websphere.objectgrid.plugins.LockDeadlockException: Message

• Problem:A LockTimeoutException exception occurs.

  Description: When a transaction or client asks for a lock to be granted for a specific map entry, the request often waits for the current client to release the lock before the request is submitted. If the lock request remains idle for an extended time, and a lock is never granted, LockTimeoutException exception is created to prevent a deadlock, which is described in more detail in the following section. You are more likely to see this exception when you configure a pessimistic locking strategy, because the lock never releases until the transaction commits.

  Retrieve more details:

  • The LockTimeoutException exception contains the getLockRequestQueueDetails method, which returns a string. You can use this method to see a detailed description of the situation that triggers the exception. The following is an example of code that catches the exception, and displays an error message.

    try {
        ...
    }
    catch (LockTimeoutException lte) {
        System.out.println(lte.getLockRequestQueueDetails());
    }

    If you receive the exception in an ObjectGridException exception catch block, the following code determines the exception and displays the queue details. It also uses the findRootCause utility method.

    try {
    ...
    }
    catch (ObjectGridException oe) {
        Throwable Root = findRootCause( oe );
        if (Root instanceof LockTimeoutException) {
            LockTimeoutException lte = (LockTimeoutException)Root;
            System.out.println(lte.getLockRequestQueueDetails());
        }
    }

  • The LockTimeoutException exception contains the getMessage method, which returns a string. You can use this method to see a detailed description of the situation that triggers the exception.

  Solution: A LockTimeoutException exception prevents possible deadlocks in your application. An exception of this type results when the exception waits a set amount of time. You can set the amount of time that the exception waits by setting the lock timeout value. If a deadlock does not actually exist in your application, adjust the lock timeout to avoid the LockTimeoutException.

  For more information about setting the lock timeout value, see Configuring the lock timeout value in the ObjectGrid descriptor XML file. You can also configure the timeout value programmatically:

  • Configuring and implementing locking in Java applications
  • Configuring and implementing locking in .NET applications

  .
• Problem: A deadlock occurs on a single key.

  Description: The following scenarios describe how deadlocks can occur when a single key is accessed with an S lock and later updated. When this action occurs from two transactions simultaneously, a deadlock occurs.

  Table 1. Single key deadlocks scenario

  	Thread 1	Thread 2
  1	Start transaction	Start transaction	Each thread establishes an independent transaction.
  2	get key1	get key1	S lock granted to both transactions for key1.
  3	One of: update key1 Put key1		No U lock. Update performed in transactional cache.
  4		One of: update key1 put key1	No U lock. Update performed in the transactional cache
  5	Commit transaction		Blocked: The S lock for key1 cannot be upgraded to an X lock because Thread 2 has an S lock.
  6		Commit transaction	Deadlock: The S lock for key1 cannot be upgraded to an X lock because T1 has an S lock.

  Table 2. Single key deadlocks, continued

  	Thread 1	Thread 2
  1	Start transaction	Start transaction	Each thread establishes an independent transaction.
  2	get key1		S lock granted for key1
  3	One of: getForUpdate key1 GetAndLock key1	get key1	S lock is upgraded to a U lock for key1.
  4		get key1	S lock granted for key1.
  5		One of: getForUpdate key1 GetAndLock key1	Blocked: T1 already has U lock.
  6	Commit transaction		Deadlock: The U lock for key1 cannot be upgraded.
  7		Commit transaction	Deadlock: The S lock for key1 cannot be upgraded.

  Table 3. Single key deadlocks, continued

  	Thread 1	Thread 2
  1	Start transaction	Start transaction	Each thread establishes an independent transaction
  2	get key1		S lock granted for key1.
  3	One of: getForUpdate key1 GetAndLock key1		S lock is upgraded to a U lock for key1
  4		get key1	S lock is granted for key1.
  5		One of: getForUpdate key1 GetAndLock key1	Blocked: Thread 1 already has a U lock.
  6	Commit transaction		Deadlock: The U lock for key1 cannot be upgraded to an X lock because Thread 2 has an S lock.

  If the ObjectMap.getForUpdate is used to avoid the S lock, then the deadlock is avoided:

  Table 4. Single key deadlocks, continued

  	Thread 1	Thread 2
  1	Start transaction	Start transaction	Each thread establishes an independent transaction.
  2	One of: getForUpdate key1 GetAndLock key1		U lock granted to thread 1 for key1.
  3		One of: getForUpdate key1 GetAndLock key1	U lock request is blocked.
  4	One of: update key1 Put key1	<blocked>
  5	Commit transaction	<blocked>	The U lock for key1 can be successfully upgraded to an X lock.
  6		<released>	The U lock is finally granted to key1 for thread 2.
  7		One of: update key2 Put key2	U lock granted to thread 2 for key2.
  8		Commit transaction	The U lock for key1 can successfully be upgraded to an X lock.

  Solutions:

  • Use the getForUpdate or GetAndLock method instead of a get method to acquire a U lock instead of an S lock.
  • Use a transaction isolation level of read committed to avoid holding S locks. Reducing the transaction isolation level increases the possibility of non-repeatable reads. However, non-repeatable reads from one client are only possible if the transaction cache is explicitly invalidated by the same client.
  • Use the optimistic locking strategy. Using the optimistic lock strategy requires handling optimistic collision exceptions. (Java applications only)
• Problem: A deadlock occurs on ordered multiple keys.

  Description: This scenario describes what happens if two transactions attempt to update the same entry directly and hold S locks to other entries.

  Table 5. Ordered multiple key deadlock scenario

  	Thread 1	Thread 2
  1	Start transaction	Start transaction	Each thread establishes an independent transaction.
  2	get key1	get key1	S lock granted to both transactions for key1.
  3	get key2	get key2	S lock granted to both transactions for key2.
  4	One of: update key1 Put key1		No U lock. Update performed in transactional cache.
  5		One of: update key2 Put key2	No U lock. Update performed in transactional cache.
  6.	Commit transaction		Blocked: The S lock for key 1 cannot be upgraded to an X lock because thread 2 has an S lock.
  7		Commit transaction	Deadlock: The S lock for key 2 cannot be upgraded because thread 1 has an S lock.

  You can use the ObjectMap.getForUpdate method to avoid the S lock, then you can avoid the deadlock:

  Table 6. Ordered multiple key deadlock scenario, continued

  	Thread 1	Thread 2
  1	Start transaction	Start transaction	Each thread establishes an independent transaction.
  2	One of: getForUpdate key1 GetAndLock key1		U lock granted to transaction T1 for key1.
  3		One of: getForUpdate key1 GetAndLock key1	U lock request is blocked.
  4	get key2	<blocked>	S lock granted for T1 for key2.
  5	One of: update key1 Put key1	<blocked>
  6	Commit transaction	<blocked>	The U lock for key1 can be successfully upgraded to an X lock.
  7		<released>	The U lock is finally granted to key1 for T2
  8		get key2	S lock granted to T2 for key2.
  9		One of: update key2 Put key2	U lock granted to T2 for key2.
  10		Commit transaction	The U lock for key1 can be successfully upgraded to an X lock.

  Solutions:

  • Use the getForUpdate or GetAndLock method instead of the get method to acquire a U lock directly for the first key. This strategy works only if the method order is deterministic.
  • Use a transaction isolation level of read committed to avoid holding S locks. This solution is the easiest to implement if the method order is not deterministic. Reducing the transaction isolation level increases the possibility of non-repeatable reads. However, non-repeatable reads are only possible if the transaction cache is explicitly invalidated.
  • Use the optimistic locking strategy. Using the optimistic lock strategy requires handling optimistic collision exceptions.(Java applications only)
• Problem: A deadlock occurs from an out of order U lock
  Description: If the order in which keys are requested cannot be guaranteed, then a deadlock can still occur.

  Table 7. Out of order with U lock scenario

  	Thread 1	Thread 2
  1	Start transaction	Start transaction	Each thread establishes an independent transaction.
  2	One of: getForUpdate key1 GetAndLock key1	One of: getForUpdate key2 GetAndLock key2	U locks successfully granted for key1 and key2.
  3	get key2	get key1	S lock granted for key1 and key2.
  4	One of: update key1 Put key1	One of: update key2 Put key2
  5	Commit transaction		The U lock cannot be upgraded to an X lock because T2 has an S lock.
  6		Commit transaction	The U lock cannot be upgraded to an X lock because T1 has an S lock.

  Solutions:

  • Wrap all work with a single global U lock (mutex). This method reduces concurrency, but handles all scenarios when access and order are non-deterministic.
  • Use a transaction isolation level of read committed to avoid holding S locks. This solution is the easiest to implement if the method order is not deterministic and provides the greatest amount of concurrency. Reducing the transaction isolation level increases the possibility of non-repeatable reads. However, non-repeatable reads are only possible if the transaction cache is explicitly invalidated.
  • Use the optimistic locking strategy. Using the optimistic lock strategy requires handling optimistic collision exceptions.(Java applications only)