Open Mic Webcast Replay: Troubleshooting SMTP inbound and outbound issues- 14 February 2012
IBM hosted an Open Mic webcast with Lotus Development and Support Engineers on 14 February 2012 on the topic of "Troubleshooting SMTP inbound and outbound issues." The Q&A transcript and recording are now posted.
IBM hosted an Open Mic webcast with Lotus Development and Support Engineers on 14 February 2012 on the topic of "Troubleshooting SMTP inbound and outbound issues." This document includes a list of questions and answers from the session, as well as the recording of the call and the presentation.
For more information about our Open Mic webcasts, visit the IBM Collaboration Solutions Support Open Mics page.
There are also some questions and answers on this topic posted to the Notes/Domino forum in this forum post.
Questions and Answers:
Q. We host about 20 different domains, and currently all users can receive email from any of the domains via a short name or alias. How can I restrict this to only a single or smaller subsets of same (they are hosted in the same Domino Domain/Directory)?
A. So you want firstname.lastname@example.org to not receive mail sent to email@example.com, where domain2 is set as an alias in the Global Domain Document?
Q. Correct.. Currently all our SMTP domains are listed on the Global Domain doc. I think I may need to break these out individually, but am not sure. In other words, we have one local primary domain, and all of the other domains listed in the Alternate Aliases area.
A. Create a separate Global Domain document for each Internet Domain you have. Also, enable Full name only instead of Full name and local part under the Configuration document > Router / SMTP > Basics. Also, be sure to cycle the SMTP server whenever you make changes to the Global Domain doc.
Q. What is the most common business case to put the SMTP task on another machine, such as SendMail? Seems like most SMTP tasks are handled natively by Domino just fine.
A. SMTP will accept anything that is intended to be sent to the domain that the Domino server is configured. We are lenient with what we receive, and strict with what we send.
Q. We occasionally get incoming messages held in the mail.box for no apparent reason. Running a "route *" command delivers the message, but I'm at a loss why this sometimes happens.
A. Need more information. Are these messages intended for local delivery, or are they being routed to the Internet? Messages going out to the Internet are placed into a Retry state, if there's issue attempting to send the message. We retry 15 minutes, 30 minutes, and then every 45 minutes for 24 hours (by default). This is also configurable.
Q. We have one domain, test.com, and many subdomains, xx.test.com, that are set up as alternate domains in the Global Domain document. Is there any helpful reason to set up different Global Domain documents for each subdomain?
A. This should be fine. However, don't use wild card subdomains, for example, *.test.com, as this does not work. Each subdomain must be defined.
Q. So all Domino servers should have two mail.boxes?
A. Domino mail servers should have at least two mail.boxes and, depending on hardware, you may want even more than that. Refer to IBM Support Technote #:1148438: " Determining the number of mailboxes required for a server."
Q. Can we use DNS names in these fields?
A. Not sure which fields you are referring to; however, most fields support both IP addresses and DNS names. An IP address within square brackets avoids DNS lookups and uses the IP value within the square brackets. Also, if you mouse-over the field, Help will usually pop up and tell you exactly what you can use.
Q. For Domino servers that relay SMTP mail from internal non-Domino systems to Domino mail users, do you have any recommendations for gathering statistics on which systems are sending SMTP mail through Domino and the volume of mail that they send?
A. Reports, Message Tracking, Stats, and Logs can help, but we do not have tools that will specifically obtain the information you refer to.
Q. Any work in progress to allow email sent from a Notes client to a Traveler user to allow body of email to be seen only by the recipient when prevent copying is selected? Currently the Traveler client is treated as a forward and the body does not show on the mobile device.
A. We have an open requirement for it ( #BWIR-87Z53L), though currently it is not on any plan for this year. You can open a PMR with Support, if you haven't already, to give the enhancement request more weight..
Q. Is there a way to prevent faked sender addresses? The sender exists in the directory but was faked and sent to the SMTP server from an internal network.
A. You could use the RouterUseFromAsSMTPOriginator=1 parameter in the Notes.ini to prevent faked sender addresses. Refer to Technote #1089673, " Can the Domino server control the Internet mail address from the server?" for more information
Q. Even if the faked sender exists in the Domino directory? What exactly does this parameter do?
A. Yes and no. The SMTP protocol allows exactly what you describe. You can ask for authentication to connect to SMTP but this does not make sense to an SMTP gateway since no users outside your domain could authenticate. The Notes.ini parameter might help, but the open nature of SMTP just allows these types of transfers.
Q. We have four Domino SMTP gateways in our Domino environment. Is there a way to load balance the outbound messages through all four systems? Can we put a load balancer within Domino? Using a third-party load balancer should work, but are there any "Domino-internal" possibilities?
A. Domino does not have a built-in load balancer. The closest option is through round-robin for SMTP. Internally you could use a Connection document to an SMTP cluster name from Domino spokes in a different NNN. There is an enhancement request for this functionality. Open a service request with IBM Support to log an additional request for this feature under SPR #SSHE5B8TQR.
Q. Outbound mail: When I send a mail message to an internal user, "John Doe", it routes it internally using NRPC, but when I send an email to the same user at "firstname.lastname@example.org", it goes out through our SMTP gateway and comes back in as an SMTP-routed message. How do I ensure that the email address is checked and is routed internally using NRPC? I would prefer that email to an Internet address be routed internally using NRPC instead of going out to the SMTP gateway, to the Internet, and then come back in.
A. By default Domino does not allow SMTP internal transfers, but that can be changed via the Configuration document. There could be things we can do, but I would need to look into your Configuration doc. Do you have a Global Domain Document? Is it configured correctly? Refer to Technote #1297369, " Understanding the Lotus Domino Global domain document." If this does not help, I would recommend opening a PMR so we can look into this deeper.
OTHER RELEVANT TECHNOTES
Technote #1568008: " Knowledge Collection: Common Lotus Notes/Domino Mail Routing problems"
Technote #1108352: " How do you configure Lotus Domino for secure SMTP sessions using the STARTTLS extension?"
Q. No promises, but feel free to post enhancement requests and business/admin justifications to the Chat or Forum, such as the outbound load balancing. How would it benefit you? What problems are you having?
- Traveler deployments. Especially for issues with encrypted mails would be nice. Having problems with that currently.
- Would also be great to have this sort of [Troubleshooting Open Mic] topic with the Traveler team.
Please allow load balancing outbound to multiple SMTP gateways from an NRPC environment. We have a very large environment that has 4 domino SMTP gateways. Inbound is load balanced, but outbound is stuck through only one of the servers. This will help with load balancing and any potential DR situation where a Domino SMTP gateway might be down due to failure or maintenance.
- I strongly agree with the earlier comments about being able to load balance. We have two outbound SMTP gateways and two inbound. We have a switch that load-balances email on the inbound route but all Domino servers only talk to the first outbound SMTP server in the Domino Directory. So the second one only gets used when the first one is down. We would prefer to run these live-live so we can be confident we will not have any issues when one or the other fails over and to maximize throughput under normal circumstances.
I'm not sure who on the panel was talking about using Connection docs and NNNs to do load balancing. That's not a very elegant solution. I appreciate Domino doesn't do load balancing right now, but I think there are a lot of customers who would like to see it, based on the comments on the chat and questions raised on call, so it would be good of IBM to consider how this could be introduced to the product - and I appreciate it's not going to be easy.
- Any thoughts about supporting DKIM signing?
- Can we trim all input fields on the Config docs as well? Had an issue 8.5.2 where a space was appended at the end of an alias when setting up a foreign SMTP doc. Took way too much time to figure that out.
Q. How can I verify the status of TLS (transport-layer security), i.e, whether I have the correct certificate, etc?
A. We have an excellent Technote, #1108352, " How to configure Domino for secure SMTP sessions using STARTTLS," on how to set up and configure transport-layer security (TLS). If you are having problems with certain domains, then open a PMR with Support to investigate further. Meanwhile I recommend your using the Notes.ini parameter, RouterFallbackNonTLS=1, for sending out TLS messages to domains. So, if receiving systems are not properly configured, or for whatever reason you're having a problem encrypting the transaction between SMTP servers, you will still be able to get your mail off your server via SMTP unencrypted.
Q. OK, so is there a quick thing I can do with telnet that can show me what my TLS settings are on my servers?
A. There are some third-party tools for TLS, but I can't recall them off the top of my head. Typically, for troubleshooting certificates, for example, I'd go to https and then either local host for your server or the receiving domain's server to see if there's a problem with the certificate. And for Domino, we use the same certificate for everything--HTTP, POP, SMTP, etc.
A. Also, one thing to make sure that your "advertising" that you accept mail over TLS, you should telnet to the server, using EHLO, which will show you the extensions that are used, including you should see STARTTLS. It won't check the certificates, but at least you'll see that your server is advertising your using TLS---or to the server to which you're trying to connect. Another thing to note is that when we say "TLS and Domino", it's actually negotiated SSL, not TLS.
Q. We're running a Domino 8.5.1 environment and have a Routing profile set up within the Domino primary Address Book to forward mail to our exchange domain as well. We are getting a number of Delivery Failure Reports (DFRs) for people who are using Notes 8.5.1 because of the Recent Contacts caching. We are trying to determine how extensive the problem is. Are there any statistics or something else we should look at to determine how many DFRs are being generated on each mail server. I did do a "showstatmail" and saw a mail transfer failure count, but that could be any type of failure. So, is there anything I could be looking at?
A. Unfortunately I don't know of any statistics that will differentiate whether a failure is specifically due the Recent Contacts.
Q. So the mail transfer failures are totally DFRs, correct?
A. Yes, it's just the overall failures, not specific to any one type of failure. Note that there is a statistic, Mail.reports.failure, that gives the the number of failure reports generated but, again, it doesn't indicate what the reasons are.
Q. Regarding load balancing outbound mail to gateways: If we're using SMTP between mail servers or other servers in Domino SMTP gateways, we can do it via load balancing but if we use NRPC , then are we out of luck?
A. Yes, there's no true way to do load balancing within Domino. We do not have a built in load balancer. Usually when asked this, I suggest using separate Notes Named Networks (NNN) and using Connection docs to the cluster name from the spokes or hub. but again, it's more a case of "pre-meditated" load-balancing using round robin. There's nothing in Domino that will come in and split things up based on traffic. There are 3rd-party load balancers that you may consider.
Q. Would you recommend a mixed environment of NRPC and SMTP where SMTP from these servers to Domino gateways, or is that not recommended?
A. Whatever works better for you. Either would work fine; generally, we recommend using NRPC between the servers internally, You can also loose functions that rely on NPRC routing, but there's no restriction on using SMTP.
Q. We have certain servers ostensibly identified as SMTP Outbound and others as SMTP Inbound. I'd like to get a rough idea of the kind of throughput---obviously this will be based on the boxes themselves---in each of the boxes. Is NRPC to SMTP conversion and SMTP to NRPC conversion roughly similar, so we should be able to get similar amounts of mail through?
In other words: If we've got, for instance, and SMTP Outbound server and if it were able to process 10K emails, therefore converting NRPC to SMTP, could we similarly expect the server to able to process roughly the same amount of emails (10K) via SMTP to NRPC? Is that conversion workload similar or are they completely different in terms of the loads they place on the server?
A. I think you're talking about a compound document to a MIME conversion. Typically when you're sending messages out to the Internet, there's a setting on the client to send Internet messages in a MIME format. In that way, there wouldn't be any conversion of a message that's routing out via SMTP to the Internet. And in the Person doc by default setting "Keep in sender's format", so that when receive a message in from the Internet it will be preserved and sent to the end user in MIME format, so there's no conversion Inbound. The only way it would be converted if the setting "Prefers Notes rich text" is enabled; then the msg. would be converted from MIME to a compound document for that user. So, by default the message is set to MIME both outbound and inbound.
Q. OK, so provided we're not doing any of that translation, it should be roughly the same.
A. Yes, and if you have users who are using third-party clients, it's recommended that their preference is set to "MIME". So, messages in their mail file don't need to be converted to a MIME format to render in the third-party clients.
Q. We have 48 download servers with 48 clustermates. With respect to the router, when we have a problem with primary servers, we don't have an easy way to determine whether the router is properly flowing messages to the clustermates, or even if that's possible.
A. We can certainly increase the level of debugging so we can see what the router is doing. We've got debugrouter=3, and we have log_mailrouting=40, which are two dynamic .INI settings that show what the router's doing. Also, we could do a "tell router show" to determine the state of the messages are with respect to the router.
Q. OK, the problem is that I have so many servers and usually it's just one server that's down; I do see where the router seems to back up messages and they're not flowing to the clustermate, Is there something inline I could look at while this is happening to see if failover is occurring?
A. Yes, "tell router show". Also, you can enable the.INI setting, routerdebugclusterfailover.
Q. When we enable Directory Assistance (DA) for LDAP authentication with iNotes, we see an issue in that SMTP is going out trying to do a handshake with our LDAP server. Is there a way to disable that, so it doesn't attempt this handshake? We talked to Support, who said we need to have our LDAP keys locally on our Domino Directory, but we are unable to do that. We feel that there should be a way to disable it, and we told that this would be addressed in Domino 8.5.3, but we're running that version and the issue has resurfaced.
A. So, if I understand this correctly: Your SMTP server is going out to your LDAP Directory to resolve incoming messages?
Q. Yes, but it's not supposed to: We put our LDAP Directory in our Directory Assistance database for HTTP authentication for iNotes, but since doing that, we're getting SMTP hangs and the call stacks indicate SMTP is attempting to handshake with our LDAP server.
A. For straightforward LDAP, I'd need to refer you to our Web Server team who support that. But the router SMTP will use its primary address book and anything that's available in the DA database to try to resolve incoming messages and even if DA is not available, there's a chance it will accept msgs for people that don't exist because of directory availability. The ability to prevent the SMTP task from looking up to the LDAP directory is something I am not aware of, and would have to refer that to the Web Server team to see if there's a fix for it. This is something that needs more investigation, but typically Domino will use any and all directories to it, to resolve addresses of incoming message. I think this would require a PMR so we could examine this further.
Q. Regarding security on Inbound controls, when we try to stop traffic to the Domino machine using Inbound connection controls allowing only from specific domains or IP addresses. This cuts down the traffic; however, we have a problem with mobile users using SMTP to send mail via POP3 client; having very very different IPs coming up, and not from providers on a specific domains so we can't identify them. Any policies that would work with that?
A. Notes Traveler would be good alternative for mobile devices. Thing is: When locking down SMTP servers to only allow traffic from specific ranges or IP addresses to connect to you, these mobile devices can be coming in from any address; using DHCP, we'll never know what they're coming in under. So, that just adds to the complexity here. Traveler is an excellent alternative as it allows synchronization of a hand-held device directly with the mail file.
Q. That is why as far as possible we we go and it's fine, but still have some users that can use this. So, no chance.
A. Is it so that these clients can send mail? Is that the idea?
Q. Yes, they want to send mail using the infrastructure.
A. OK, well, POP3 and IMAP you pull messages down from the server and can actually configure these clients to send mail from the carrier/ISP they're on and that should be a viable solution. So they wouldn't need to worry about where msgs are coming in from. the connections
Q. We see with at least one, we get not very specific inbound addresses from the relay servers used; e.g., a network in Germany seems to send mail under a wide variety of IP addresses.
A. Yes, that's a tough one, when you're limiting who you get connections from for authenticating to send mail.
Q. So is it best to stay with a very strict usage of "Allow connections from", to keep traffic on the server down, and increase enforcement of people to use Traveler, IMAP, etc?
A. Well that depends. Do you have a third-party filter out in front of your server, such as Postini; in that case, you'd just restrict your inbound mail so you only get mail from Postini, and others couldn't connect. If Domino is to be your Gateway, then you'd want to lock down that gateway as much as possible. We've long cautioned against using SMTP authentication because we've seen a number of cases in which customers were hacked into where usernames and password dictionary attacks were conducted against servers to get in, to be able to use the Domino server as their own relay, so I would not recommend enabling "STMP AUTH ". Instead I'd go back to the carriers of the hand-held devices to see if they could provide an SMTP relay. And you'd still be able to pull down your mail via POP3 or IMAP.
Q. We have an issue with SMTP outbound. When there's a bad message that Domino can't transfer, it creates a queue of msgs pending. SMTP inbound continues to work, router continues to work, but SMTP outbound queue gets built up. Do you have any means--a parameters maybe--that can do a load balancing to send these messages to the other SMTP server or to open other threads to transfer the rest of the messages?
A. That used to be a problem with Domino versions earlier than 7.0.3.
Q. We haven't had the problem in a long time, and we upgraded Domino to 8.5.3 recently and the problem reoccurred. It's happened three times in two months, all with Japanese emails containing Japanese characters. When we remove these messages from mail.box, everything works fine. We opened a PMR with IBM but they couldn't reproduce it.
A. Are you sending directly to the Internet or using a relay host?
Q. We use the IronPort device.
A. That's probably why we couldn't reproduce it; we were sending directly out to Internet.
Q. I even tried it in a test environment that is the same way, and mail go out fine directly to Internet. Debugging revealed that it's getting stuck after the data is sent, and then the reports says that Domino closes the connection.
A. Have you opened a ticket with IronPort?
Q. No, but I cannot reproduce the issue with IronPort on. If I move the message on, it will transfer. I can place the message back into queue, and there's no problem. So the problem is with Domino. Are there additional threads that can handle the other messages?
A. Absolutely. Again, since 7.0.3, if there's a problem email, it's pushed aside and other mail continues to route behind it. That's why I asked if you opened a ticket with I-import because Domino is completely able to create separate queues for an email that we cannot move on, and process the rest of the mail behind them.
Q. Well, I-import is able to handle multiple message and multiple connections, so we don't have limits on connections from the Domino servers.
A. When you get down to the data portion of the message, there may be something that IBM is not seeing, and you may need a sniffer to look at the communication between the two applications to understand why I-import is not accepting that message. In any case, this issue needs a PMR.
Q. Our PMR is still open, and I have copies of the msgs. Other thing: We have two internal Domino servers, and all the SMTP mail goes to the server listed first; is there any way to load balance this so mail goes to the other server?
A. Again, Domino does not have built-in load balancing. The closest thing to this in a Notes environment is by creating a Connection document to the cluster name. The cluster gateway would have to be in its own Notes Named Network and basically, the hubs and spokes behind that cluster, you would have a Connection doc that cluster name and then you'd have a round-robin-type effect in which the servers would pick up that connection and handle the msg.
Q. Regarding a specific message we saw when sending out mail to a company from a Domino server, we got the information, "Reported responsible address failure," error from the receiving company. Problem is that we can send mail from that same Domino environment to other users via that same address in that same company, but only from a specific mailbox; otherwise, we get the error every time. Any idea why?
From RFC 4407 from 2006, it seems to be related to the sender's From address recognition that the receiving party acknowledges this as a failure.
A. In this case, the first thing I'd do is put on a debug parameter, SMTPClientDebug=1 (and zero to disable), to look at the Outbound SMTP conversation and have the user who's having the problem send another message. Then look at your console.log or in Misc Events view of your Log.nsf, try searching for this conversation, and see what error's reported. Sometimes the log shows a more detailed error msg than in the Delivery Failure itself. From what you've told us, it sounds like something on the receiving end is rejecting you; the debug will provide more detail and we can figure out how to proceed. Also, try using telnet from the sending server.
Q. We can send mail from the sending server if it goes out from other users at that address on that Domino server, so it's one specific user on one specific mail.box, which by the way, is not a standard Notes mailbox.
A. You could step through the telnet commands with the mail from your sender and the "Receipt to" that recipient, and maybe get more information from the telnet communication with that server, along with using the SMTPClientDebug parameter that Sarah mentioned. Another thing: I've seen a case in which a period was appended to the domain, or something was slightly different that was not easily noticed that came from Recent Contacts, so you want to check for that as well.
Q. I saw another thing in that box that could be a problem; there maybe were several "Froms" in the To mail document and we don't know where that came from. So the From address is normally that one in the Domino item environment, and then we had a forwarded From as another item and have no idea how Domino translates that or where it comes from.
A. Domino builds a msg based on several factors. For msgs going out, it looks at the ID and pulls info from the ID, the Location doc of the client, the Internet address field of the Person doc, and the Global Domain doc. Also it looks the Configuration doc to look up Internet addresses for all Notes addresses leaving the local Internet domain. so that's something that should be enabled. Basically, if the Global Domain doc can't look up the Internet address, it builds one with what you have set on the Conversions tab. So as long as everything is properly configured, Domino will do its best to build a good, reply-able address.
Q. OK, so my next step is to see what's going on in the conversation between the servers, and then look at the Console output? OK, thanks!
Original publication date