Users stop receiving mail and SMTP task on server shows 50-100% CPU utilization

Technote (troubleshooting)


Problem

After upgrading to Lotus Domino 8.5.2, users stop receiving SMTP mail. Your Domino server CPU spikes on SMTP and may appear to be hung.

The "show tasks" command may show the following thread. You may see numerous SMTP threads or a single SMTP thread:

Server.Task = SMTP Server: Providing service for 123.45.678.910
Server.Task = SMTP Server: Providing service for 123.45.678.911
Server.Task = SMTP Server: Providing service for 123.45.678.912

nSMTP can spike the CPU anywhere from 50% to 100%.

This is caused by a specific SMTP message where we see 50-100% CPU, depending on the platform.

Restarting the Domino server may resolve the hung task temporarily, but it is possible that another message will get stuck and the CPU will spike and appear to be hung again.

Resolving the problem


    This issue is now fixed in Lotus Notes and Domino 8.5.2 FP1 and 8.5.3.

    Fix details: SPR# KCAR8A7NB3

    Refer to the Upgrade Central site for details on upgrading Notes/Domino.

    To diagnose this problem on Windows you will need to perform a series of necessary data captures as follows:

    1. Ensure the following Debug is enabled on your Server

    Console_Log_enabled=1

    Debug_Threadid=1

    SMTPDebugIO=3

    SMTPSaveImportErrors=3

    The message is written to a file in the system Temp directory with a name STXXXXXX.TMP. When a message is saved in the temp file, a line indicating the name of the file will be written to the log. The value of the parameter controls the saving of messages as follows:

    2. From the Server Console, issue a Show Tasks command. You may see numerous SMTP Threads or a single SMTP thread

    Server.Task = SMTP Server: Providing service for 123.45.678.910
    Server.Task = SMTP Server: Providing service for 123.45.678.911
    Server.Task = SMTP Server: Providing service for 123.45.678.912


    3. A screen capture of the Windows Task Manager showing the nSMTP Task taking 50% CPU up to 100%:




    4. Run a Manual NSD to capture the data set. Running a manual NSD can take a long time as the CPU is consumed by the SMTP task; however, this is essential in determining what the thread is doing. You should see a stack similar to the following:

    Note: How to run a manual NSD for Notes/Domino on Windows


    ############################################################
    ### thread 4/6: [ nSMTP: 0cc0: 17a8]
    ### FP=0x150dd7e8, PC=0x600013e6, SP=0x150dd7ac
    ### stkbase=0x150e0000, total stksize=262144, used stksize=10324
    ############################################################
    @[ 1] 0x600013e6 nnotes.OSUnlockSem@4+54 (15bb72b4)
    @[ 2] 0x601df473 nnotes.UnlockMemHandle@4+19 (15bb728c)
    @[ 3] 0x60002c31 nnotes.OSUnlockObject@4+129 (611e9090)
    @[ 4] 0x6001b68c nnotes.NSFAddToNameLookupTable@12+268 (15c22208,1ed,611e7718)
    @[ 5] 0x6001dd2a nnotes.ItemAppendByBLOCKID@32+570 (150dd8b4,4,60eebc54,4,0,150dd870,f10f10,ffffffff)
    @[ 6] 0x60048ecd nnotes.NSFItemAppendByBLOCKID@32+93 (59,3,60eebc54,15c22110,0,150dd8a0,f10f10,ffffffff)
    @[ 7] 0x600522e0 nnotes.NSFItemAppend@28+160 (59,3,60eebc54,4,0,150dd8dc,f10f10)
    @[ 8] 0x60358756 nnotes.NSFMimePartAppendTG@36+182 (59,60eebc54,4,2,0,150dd928,f10f10,ffffffff,0)
    @[ 9] 0x60351e29 nnotes.MIMEItemizeMessageBodyItemAppend+2393 (150deb30,59,60eebc54,4)
    @[10] 0x603527da nnotes.MIMEItemizeMessageBody+1162 (150deb30,59,60eebc54,4)
    @[11] 0x603522d4 nnotes.MIMEItemizeMessage+804 (150deb30,59,60eebc54,4)
    @[12] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (150deb30,59,60eebc54,4)
    @[13] 0x603522d4 nnotes.MIMEItemizeMessage+804 (150deb30,59,60eebc54,4)
    @[14] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (150deb30,59,60eebc54,4)
    @[15] 0x603522d4 nnotes.MIMEItemizeMessage+804 (150deb30,59,60eebc54,4)
    @[16] 0x60352a74 nnotes.MIMEItemizeUsingCallbackExt@32+372 (59,60eebc54,4,7,0,150deb40,f10f10,ffffffff)
    @[17] 0x60352cad nnotes.MIMEItemizeStreamExt@28+93 (59,0,4,7,0,150dee78,f10f10)
    @[18] 0x603531a2 nnotes.MIMEItemizeStream@24+34 (59,0,0,7,0,150dee9c)
    @[19] 0x602b24ed nnotes.CIMsgImport::Import+221 (14f513b8,397030,59,3)
    @[20] 0x004136d3 nSMTP.CSMTPProtocol::SubmitMessage+467 (14f513b8,150dfc6a,2,14648018)
    @[21] 0x0041192f nSMTP.CSMTPProtocol::CommandDATA+1407 (8018,2,14648018,0)
    @[22] 0x00412bb6 nSMTP.CSMTPProtocol::StateConnected+1158 (14648018,390000,397010,397008)
    @[23] 0x00412ecd nSMTP.CSMTPProtocol::Run+429 (14648018,128de6c,43bf68,128de6c)
    @[24] 0x0042183e nSMTP.CBaseTask::StateMachine+398 (397008,128de6c,43bf68,128de88)
    @[25] 0x00402fe3 nSMTP.CSMTPSrv::OnConnect+211 (43bf68,128de88,13d80001,128de6c)
    @[26] 0x004173a6 nSMTP.CIServ::ServerTaskProtocolMachine+262 (43bf68,128de6c,1232ee8,12357f0)
    @[27] 0x0041bd9a nSMTP.CIServ::ServerTaskIOCP+922 (43bf68,0,6015584f,0)
    @[28] 0x0041c71d nSMTP.ServerThread@4+29 (0)
    [29] 0x77e6482f kernel32.GetModuleHandleA+223 (601557a0,0,0,dddd04d2)

    we also saw this stack:

    ############################################################
    ### thread 7/17: [ nSMTP: 052c: 0cc8]
    ### FP=0x1978c054, PC=0x600013e6, SP=0x1978c018
    ### stkbase=0x19790000, total stksize=262144, used stksize=16360
    ############################################################
    @[ 1] 0x600013e6 nnotes.OSUnlockSem@4+54 (18ff3808)
    @[ 2] 0x601df473 nnotes.UnlockMemHandle@4+19 (18ff37e0)
    @[ 3] 0x601df5cf nnotes.OSLockObjectExt@8+143 (611e9090,611e9142)
    @[ 4] 0x60002b7e nnotes.OSLockObject@4+14 (94)
    @[ 5] 0x6001b694 nnotes.NSFAddToNameLookupTable@12+276 (a40a08,1ef,611ed988)
    @[ 6] 0x6001dd2a nnotes.ItemAppendByBLOCKID@32+570 (1978c138,4,60eebc54,4,0,1978c0f4,f10f10,ffffffff)
    @[ 7] 0x60048ecd nnotes.NSFItemAppendByBLOCKID@32+93 (83,3,60eebc54,a40910,0,1978c124,f10f10,ffffffff)
    @[ 8] 0x600522e0 nnotes.NSFItemAppend@28+160 (83,3,60eebc54,4,0,1978c160,f10f10)
    @[ 9] 0x60358756 nnotes.NSFMimePartAppendTG@36+182 (83,60eebc54,4,2,0,1978c1ac,f10f10,ffffffff,0)
    @[10] 0x60351e29 nnotes.MIMEItemizeMessageBodyItemAppend+2393 (1978d3b4,83,60eebc54,4)
    @[11] 0x603527da nnotes.MIMEItemizeMessageBody+1162 (1978d3b4,83,60eebc54,4)
    @[12] 0x603522d4 nnotes.MIMEItemizeMessage+804 (1978d3b4,83,60eebc54,4)
    @[13] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (1978d3b4,83,60eebc54,4)
    @[14] 0x603522d4 nnotes.MIMEItemizeMessage+804 (1978d3b4,83,60eebc54,4)
    @[15] 0x603525b7 nnotes.MIMEItemizeMessageBody+615 (1978d3b4,83,60eebc54,4)
    @[16] 0x603522d4 nnotes.MIMEItemizeMessage+804 (1978d3b4,83,60eebc54,4)
    @[17] 0x60352a74 nnotes.MIMEItemizeUsingCallbackExt@32+372 (83,60eebc54,4,13,0,1978d3c4,f10f10,ffffffff)
    @[18] 0x60352cad nnotes.MIMEItemizeStreamExt@28+93 (83,60eebc54,4,13,0,1978d6fc,f10f10)
    @[19] 0x603531a2 nnotes.MIMEItemizeStream@24+34 (83,60eebc54,4,13,0,1978d720)
    @[20] 0x602afd79 nnotes.CIMsgImport::ImportConvertDeliveryStatus+505 (19891498,83,1,1978f000)
    @[21] 0x602b0ba8 nnotes.CIMsgImport::ImportConvertHeaders+216 (19891498,c736d0,83,1978f000)
    @[22] 0x602b2587 nnotes.CIMsgImport::Import+375 (19891498,c736d0,83,3)
    @[23] 0x004136d3 nSMTP.CSMTPProtocol::SubmitMessage+467 (19891498,1978fc6a,2,19cba018)
    @[24] 0x0041192f nSMTP.CSMTPProtocol::CommandDATA+1407 (a018,2,19cba018,656e6968)
    @[25] 0x00412bb6 nSMTP.CSMTPProtocol::StateConnected+1158 (19cba018,c70000,c736b0,c736a8)
    @[26] 0x00412ecd nSMTP.CSMTPProtocol::Run+429 (19cba018,12ddd64,43bf68,12ddd64)
    @[27] 0x0042183e nSMTP.CBaseTask::StateMachine+398 (c736a8,12ddd64,43bf68,12ddd80)
    @[28] 0x00402fe3 nSMTP.CSMTPSrv::OnConnect+211 (43bf68,12ddd80,12240001,12ddd64)
    @[29] 0x004173a6 nSMTP.CIServ::ServerTaskProtocolMachine+262 (43bf68,12ddd64,1282ee8,12857f0)
    @[30] 0x0041bd9a nSMTP.CIServ::ServerTaskIOCP+922 (43bf68,0,6015584f,0)
    @[31] 0x0041c71d nSMTP.ServerThread@4+29 (0)
    [32] 0x77e6482f kernel32.GetModuleHandleA+223 (0,0,0,0)


    On AIX, after running an NSD you will see the following stack:

    ###################################
    ###### thread 4/6 :: smtp, pid=196710, tid=2089079, ptid=1029) ######
    ###################################
    [1] 0x0000000000003438 _check_lock() + 0x18
    [2] 0x0000000110ee5c8c ????????(??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??,
    ??, ??,??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??,
    ??, ??, ??,??, ?, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??,
    ??, ??, ??, ??,??, , ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??,
    ??, ??, ??, ??, ??,?,??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??) + ??
    [3] 0x09000000006ede70 UnlockHandle(0x1100ac798) + 0x1c
    [4] 0x09000000006ef054 OSLockObjectExt(0x1d0000001d, 0x0) + 0xd4
    [5] 0x09000000006ef1dc OSLockObject(0x1d0000001d) + 0x1c
    [6] 0x0900000000b4fd6c NSFAddToNameLookupTable(0x11019df08, 0x2deb38000a) + 0x218
    [7] 0x0900000000b665e8 ItemAppendByBLOCKID(0x110ee6070, 0x3000000000003,
    0x9000000028cddd8, 0x4000000000004, 0x2deaf0003a, 0x3c0000003c, 0x0) + 0x254
    [8] 0x0900000000b67430 NSFItemAppendByBLOCKID(0xa0000000a, 0x3000000000003,
    0x9000000028cddd8, 0x4000000000004, 0x2deaf0003a, 0x3c0000003c, 0x0) + 0xec
    [9] 0x0900000000b67d3c NSFItemAppend(0xa0000000a, 0x3000000000003, 0x9000000028cddd8,
    0x4000000000004, 0x19000000000019, 0x11003cce0, 0x3a0000003a) + 0x134
    [10] 0x0900000000ea9b68 NSFMimePartAppendTG(0xa0000000a, 0x9000000028cddd8,
    0x4000000000004, 0x2000000000002, 0x100000001, 0x110ee6408, 0x26000000000026,
    0x26000000000026) + 0x170
    [11] 0x090000000102f47c MIMEItemizeMessageBodyItemAppend(tagAttachContext*,unsigned
    int,char*,unsigned short,unsigned int,tagMIMEPartType,int,int,tagMimeWellKnownSymbolId,
    tagMimeWellKnownSymbolId,tagMimeWellKnownSymbolId,tagMimeWellKnownSymbolId,CMemStr&,C
    MemStr&,CMemStr&,CStream*,CMemStr&)(0x110ee8790, 0xa0000000a, 0x9000000028cddd8,
    0x4000000000004, 0x1200000012, 0x200000002, 0x100000001, 0x100000001) + 0xe48
    [12] 0x090000000102fa7c MIMEItemizeMessageBody(tagAttachContext*,unsigned int,char*,unsigned
    short,unsigned int,CMIMEReadBuffer&,CMimeParse&)(0x110ee8790, 0xa0000000a,
    0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7200) + 0x364
    [13] 0x090000000102d108 MIMEItemizeMessage(tagAttachContext*,unsigned int,char*,unsigned
    short,unsigned int,CMIMEReadBuffer&,CMimeParse&,unsigned int*)(0x110ee8790, 0xa0000000a,
    0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7200, 0x0) + 0x4c8
    [14] 0x090000000102f988 MIMEItemizeMessageBody(tagAttachContext*,unsignedint,char*,unsigned
    short,unsigned int,CMIMEReadBuffer&,CMimeParse&)(0x110ee8790, 0xa0000000a,
    0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7b90) + 0x270
    [15] 0x090000000102d108 MIMEItemizeMessage(tagAttachContext*,unsigned int,char*,unsigned
    short,unsigned int,CMIMEReadBuffer&,CMimeParse&,unsigned int*)(0x110ee8790, 0xa0000000a,
    0x9000000028cddd8, 0x4000000000004, 0x1200000012, 0x110ee84d0, 0x110ee7b90, 0x0) + 0x4c8
    [16] 0x090000000102f988 MIMEItemizeMessageBody(tagAttachContext*,unsignedint,char*,unsigned
    short,unsigned int,CMIMEReadBuffer&,CMimeParse&)(0x110ee8790, 0xa0000000a,
    0x9000000028cddd8, 0x4000000000004, 0x1200000013, 0x110ee84d0, 0x110ee8510) + 0x270
    [17] 0x090000000102d108 MIMEItemizeMessage(tagAttachContext*,unsigned int,char*,unsigned
    short,unsigned int,CMIMEReadBuffer&,CMimeParse&,unsigned int*)(0x110ee8790, 0xa0000000a,
    0x9000000028cddd8, 0x4000000000004, 0x1300000013, 0x110ee84d0, 0x110ee8510, 0x0) + 0x4c8
    [18] 0x090000000102c2e8 MIMEItemizeUsingCallbackExt(0xa0000000a, 0x9000000028cddd8,
    0x4000000000004, 0x1300000013, 0x9001000a0781240, 0x110ee88b0, 0x0, 0x0) + 0x178
    [19] 0x090000000102bb04 MIMEItemizeStreamExt(0xa0000000a, 0x9000000028cddd8,
    0x4000000000004, 0x1300000013, 0x110ee9760, 0x0, 0x0) + 0xc4
    [20] 0x090000000102ba04 MIMEItemizeStream(0xa0000000a, 0x9000000028cddd8,
    0x4000000000004, 0x1300000013, 0x110ee9760, 0x0) + 0x44
    [21] 0x0900000001038a7c CIMsgImport::ImportConvertDeliveryStatus(unsigned int,int,unsigned int*)
    (0x1100d2cf8, 0xa0000000a, 0x100000001, 0x110eeac44) + 0x364
    [22] 0x0900000001035880 CIMsgImport::ImportConvertHeaders(CStream&,unsigned int,unsigned
    int*,CONVERSION_CONTROLS*,int)(0x1100d2cf8, 0x1113161e8, 0xa0000000a, 0x110eeac44,
    0x700000004ee64a0, 0x100000001) + 0x11f0
    [23] 0x09000000010343c4 CIMsgImport::Import(CStream&,unsigned int,unsigned short,unsigned
    int,CONVERSION_CONTROLS*)(0x1100d2cf8, 0x1113161e8, 0xa0000000a, 0x3000000000003,
    0x2700000027, 0x700000004ee64a0) + 0x204
    [24] 0x000000010002f7b8 CSMTPProtocol::SubmitMessage(unsigned short*)(0x1100eb638,
    0x110eebcc0) + 0x308
    [25] 0x000000010002a418 CSMTPProtocol::CommandDATA()(0x1100eb638) + 0x138
    [26] 0x00000001000267c0 CSMTPProtocol::StateConnected()(0x1100eb638) + 0x25c
    [27] 0x0000000100024630 CSMTPProtocol::Run()(0x1100eb638) + 0x29c
    [28] 0x000000010001bd78 CBaseTask::StateMachine()(0x1113161b0) + 0x25b0
    [29] 0x00000001000049f0 CSMTPSrv::OnConnect(SESSIONID&,unsigned int,INETTASK*)
    (0x1100005c8, 0x1101e9068, 0x110d0000110d0, 0x1101e9038) + 0x98
    [30] 0x0000000100013f10 CIServ::ServerTaskProtocolMachine(INETTASK*)(0x1100005c8,
    0x1101e9038) + 0x5c8
    [31] 0x00000001000122c4 CIServ::ServerTaskIOCP(int)(0x1100005c8, 0x100000001) + 0x7ac
    [32] 0x0000000100011a8c ServerThread(0x0) + 0x30
    [33] 0x09000000006b6900 ThreadWrapper(0x0) + 0x118
    [34] 0x09000000004a34f4 _pthread_body(??) + 0xdc


    5. Once the above data has been collected, issue a Tell Smtp Quit command. You may see the following messages and the SMTP task does not stop. A hard stop of Domino is required:


Rate this page:

(0 users)Average rating

Add comments

Document information


More support for:

IBM Domino
SMTP / MIME

Software version:

8.5.2

Operating system(s):

AIX, AIX 64bit, Linux, Windows, Windows 64bit, i5/OS

Reference #:

1445280

Modified date:

2011-09-27

Translate my page

Machine Translation

Content navigation