Why are 'd502012c error on workspace dereference' errors being logged in the Process Engine server elog? How can the errors be corrected?
Intermittently, the Process Engine(PE) server elog, located in /FNSW_LOC/logs/elogs, will log the following entries:
2012/01/05 11:45:42.181 <S103092> VW/Process (6564.7476.64 0x19a4.1d34) ... [CRITICAL]
VW (VWKs.1) (Region=50): Internal Error: 'objId > UNDEFINED_OBJECT_ID', file='../../../src/vwperobj/objsvc.cpp', line='208'
2012/01/05 11:45:42.727 202,0,27 <S103092> VW/Process (6564.7476.64 0x19a4.1d34) ... [INFO]
fn_stack_trace: stack trace saved in file 'D:\FNSW_LOC\tmp\/pe_interr_stack_6564_5.txt'
2012/01/05 11:45:42.727 <S103092> VW/Process (6564.7476.64 0x19a4.1d34) ...
VW (VWKs.1) (Region=50): Wob#: 00000000000000000000000000000000, WP class#: 0, Error: d502012c error on workspace dereference.: file='../../../src/vwruncom/qrec.cpp', line='605'
The above error messages may also be followed by an additional "Process aborting" error message, as follows:
2012/01/05 11:45:42.727 202,0,9 <S103092> VW/Process (6564.7476.64 0x19a4.1d34) ... [CRITICAL]
Process aborting: Exception 0xc0000005 encountered by process!
This will cause the VWKs associated with the error to abort.
This is an intermittent issue, and is only likely to occur under the following conditions and configuration:
1) High concurrency, a lot of work being processed
2) Large BLOBs are involved
3) The concurrent activity involve the same Process Engine queue/table
4) The database is Microsoft SQL
In investigating this problem with Microsoft (MS), it was determined that if a database table field that is in the ORDER BY clause of a SQL query statement, and the same field is not part of the table index used in the statement, MSSQL may return a null/empty value as one of the results. When PE encounters a null/empty value from the database for a query result set, it will end up logging a 'workspace dereference' error in the PE elogs.
SELECT A,B,C,D,E,F WITH ( INDEX (Index1) )WHERE A=1 ORDER BY A,B,C,D
Where Index1 contains only the fields A,B,C. However, the ORDER BY in the SQL statement contains an additional field D, which is not part of the index.
The null/empty value is the result of a dynamic cursor change at runtime within Microsoft SQL. Here is the explanation provided by MS in regards to why a null/empty value is returned:
1. There was an Implicit Cursor conversion from Dynamic cursor to key set cursor.
2. The Key Set cursor does not reflect any rows deleted on the actual table.
3. A row was deleted between the Cursor Open and the Fetch.
4. The CWT Table (Cursor Work Table) will still hold the deleted row as per design.
5. The left outer join is triggering the rows to be returned null which is by design.
Process Engine 4.x, using Microsoft SQL Server database.
Resolving the problem
Install the fix for APAR PJ40379 if the VWKs processes are aborting due to the "workspace dereference" errors. APAR PJ40379 does not address the root cause of the "workspace dereference" problem, it does provide stability to the Process Engine server when a "workspace dereference" error has occurred. Per the explanation above in the 'Cause' section, the root cause of the issue is with the SQL statement that is being sent to the server, specifically with the index that is being used and the fields that are in the ORDER BY clause.
There are 2 possibilities where the SQL query statement being sent to the database server may have fields in the ORDER BY clause that are not in the Index.
1) For OOTB queues/tables, this may be related to the table associated with the Inbox queue. This is because an additional field was added for the F_SortRule index, which did not exist in earlier releases of PE. This field is the F_BoundUser field.
Check the F_SortRule index on the Inbox queue to see if the same fields exist in the database index.
Use vwtool > queueconfig on the Inbox queue
With the config, the actual name of the index in the database will be displayed, in the below case, it is VWqsort50_115.
After getting the actual database index name, check in the database to determine if the index has the same defined fields as is displayed in the vwtool > config output. If there are discrepancies, correct the database index to match what is defined in vwtool > config, and rebuild the index.
2) If the Inbox queue index does not have any issues, then it is possible that the problem is caused by a custom query. This will be more difficult to track down, and will need assistance from the customer in relation to what index is being used as part of the custom query and whether the custom query is sorting on a particular column. Once that is determined, it is a matter of making sure that the column being sorted on is part of the index that is being used in the custom query.
This is no longer an issue for Process Engine 5.x..