Open Mic Webcast: Troubleshooting Duplicate Mail Messages in IBM Notes - 24 July 2013 [Presentation, Recording and Q&A transcript attached]
On July 24, 2013 members of the IBM Notes Domino Support and Development team shared their tips and tricks when troubleshooting repeating meetings, busytime and resource reservations.
Attendees were given an opportunity to ask a panel of experts some questions. The call was recorded and slides were made available.
Troubleshooting Duplicate Mail Messages in IBM Notes Open Mic Jul 24 2013 (edited).mp3
Q: Do we have any script which we can run to give a separate view?
A: The script we have included in this presentation is a manual script. We suggest you print it out and go through these steps.
Q: Can you type the parameter here please?
RouterDupElimLookedUpFullName= (set to 1 or 2) RouterDupElimLookedUpFullName should not be used permanently as it will slow down the router. However, for troubleshooting duplicate messages it is fine.
Q: Do we have any script which we can run to give a separate view? Can we run it on a user mail file which will help me to delete duplicate emails?
A: No - not with Domino as it comes "out of the box".
Q: Can you say that again? How to open the email to see the content? I meant by emails in mail.box.
A: We can email it to you. - Right click in the body of the email and select Document Properties. Go to the second tab - Properties. You can copy them into a folder in your mail file. The reason you copy the emails into a folder in a mail file is because you do not have access to the properties of the message from the server's mail.box.
Q: The speaker just said we can quit router.
Q: Any folder? The speaker just mentioned junk folder.
A: Make a test folder to make it easy for you.
Q: I have a custom application that writes directly to mail.box and every once in a while when someone sends a message from this custom application it will send a duplicate. I can't really troubleshoot it with the stuff we went through today because it's sent from us to the internet and we only see one copy of the message, the copy in their sent folder. But the recipients out there on the internet gets two copies. I think you've given me some ideas, but I wonder if you could address that.
A: First thing I would do is quit the router and see how many messages are getting deposited. Is the application actually depositing more than one message. Then take a closer look at the application. Another thing is to use an INI parameter, something to the effect of SMTPSAVEDOUTBOUNDFILE, and this will capture all of the messages leaving the Domino Server and an ST or a TEMP file which is the raw MIME of the message. From that, we would be able to see if multiple messages were sent from the Domino Server. It just may be the case that we are sending one, but for whatever reason the recipient server is splitting the message up and delivering two. We have seen that as well, coming into Domino, where groups and individuals in different TO and CC fields, etc... we would find that recipients would receive duplicate messages. Looking at the messages we would see that the sending domain actually splitting up that message before sending to us. So that's another possibility.
To add to that, what I would probably do is, on the receiving end, look at the received field and look at the timstamps, and you want to see if the timestamps are the same or different. Look at the duplicate message at each "hop" from when it leaves the server. If the timestamps are the same then it's the same message. If somehere in those received fields you notice there is a difference in the timechange, that could tell us where the duplication might be occurring. It could be occurring on a firewall, antivirus scanning device, or on another relay. It may not necessarily be Domino. Another thing that I would do is enable SMTPCLIENTDEBUG and look at the smtp conversation and look to see if domino is only sending one message out or are we actually sending more than one message out. Like I said, if we only see one message leaving Domino it could be that the message is being duplicated elsewhere on a hop further on down it's destination. SMTPDEBUG does not write to the log.nsf, you would need to enable console logging. Set that value to 1. The debug is dynamic so you can use the setconfig command from the server console and you don't need to stop and restart the router task.
Since only external users in one domain are affected, and no internal users, I would definitely look at the receipt fields. If you don't have access to the external recipients' mail, then monitor the console log. You can also stop the router and see what goes into the mail.box. The receiving domain can save the messages as a .EML so you can look the headers and see where potentially the messages are being split. This may be on their server.
Q: We had an issue last week where an internal e-mail address is sending lots of e-mail messages outbound on our mail.box and I just want to know what is causing that problem.
A: I think that's something we would have to take a closer look at because there could be many reasons for your server to be sending out a lot of messages. One little trick I use when I want to see what is going on with messages, you can quit the router and there won't be any loss of data or anything like that, and then you can take as an administrator and make a copy of the messages out of the mail.box and then you can paste them into your mailfile like in the junk folder. Then you can open up the messages and see them for what they are. If those messages appear to be spam in nature, then you should give us a call and log a PMR so we can take a closer look at your configuration and what's going on and where those messages are coming from.
Q: In other words there is probably something that has been compromised on our end?
A: Potentially, one of the things that we have seen is customers using OFF on their servers, where they allow their end users to come in from the internet and authenticate with their mailfile to look for new mail, etc... or use the server as a relay outside of Domino. That can be a problem if user names and passwords are not secure because spammer/hacker can compromise those connections and use your server as his own personal spam relay. So, OFF, I wouldn't recommend at this time having that feature enabled. We have taken steps later on in the code to make OFF more secure. It may be coming from another source, where other systems are open and start sending mail out through the Domino gateway. If these messages are not normal in nature for you environment then we should investigate where they are coming from and why so we can lock your server down.
Q: Is that the only resolution, there is no patch or anything we can throw on it?
A: Not until we know what the problem is. We need to know the source of these messages, are they spam, is it an end user, what is the nature of these messages that are being sent out? Are they coming from an application, are they coming from another server that's sending out notifications to a third party or something like that? It's really difficult to tell without actually looking at the physical messages and the intent of these messages. Copying them out of the mail.box and copying them into your mailfile; you'll be able to see them for what they are.
Q: There are hundreds of these messages coming out every minute. They are both MIME and SMTP and are addressed to users not even in our domain. So it's probably something using our gateway as a relay?
A: That's possible, and you definitely don't want to be an open relay or a compromised relay because that'll get you blacklisted pretty quick. It's a lot of work to get off of those blacklists. If you have too many mail messages in the queue, it'll start to shut down your server. You can call Support and generated a PMR so we can look into this.
Q: This is a general mail routing question. We have internet users sending out e-mails to the internet address of a distribution group. This group has 1 Notes mail box address and an internet address. SO when the user is sending out from the internet we are able to receive that e-mail to the internal mailbox but we don't have any track or any information about the e-mail that needs to go out of the domain to the internet addresses. The e-mails are not being received by the internet address. We don't have any information about where these are going.
A: That's not an easy one because we do not have control over other servers out on the internet. We can see that we did transfer a message to another domain. The best parameter for that is our SMTPCLIENTDEBUG= set 1or 3, I recommend 3 to get more out of it. 1 will show us the message has been transferred to the intended recipient's domain. Once it gets to their domain it's then the responsibility of that domain to deliver it to the group. On your side if messages don't appear to be received, that would be the first thing to set to make sure that, yes, you are transferring these messages over and for whatever reason, once they get there, it's that other domain that's not delivering them to their end users.
Q: It's not happening to every message, only a couple. So SMTPCLIENTDEBUG will give us more detail with what happened to that e-mail?
A: Yes, for any message going out to the internet and to their domains. There are also parameters for inbound: SMTPDEBUGIO = 3, debug_router, and log_mailrouting if you're having an internal routing issue and recipients are not getting those messages. What you're describing sounds like it is going out to the internet and the recipients are not receiving it. I would go with SMTPCLIENTDEBUG and also console_log_enabled=1. Both of those can be set dynamically by using the SETCONFIG command. Let it run. Other customers set them and keep them on permanently so they can see what is being sent. I also want to add, that the SMTPCLIENTDEBUG does display the SMTP conversation and it will also display the server that you are connecting to the responses back that you're getting, like the 250 OK, the 354, it'll display the responses from whoever you're sending the e-mail and that you're connected to the responses that you are getting back that the Domino Server is receiving. If we're not seeing the responses back or even seeing the recipient TO or Mail FROM then we know that Domino isn't sending it. But if we see the entire conversation and the message leaves Domino and is transferred we will see it in the debug. It's also helpful for other issues if you're getting a rejection you'll be more likely to see the complete rejection such as '550 no such user' stuff like that on your console. Especially lengthy rejections from other domains that give you URLs and everything else; these will be displayed on your console.
Original publication date
|Messaging Applications||IBM Domino||Mail Server|