« Troubleshooting 101: Exchange Outbound E-mail | Main | Backup, Backup, Backup. Did I mention Backup? »
February 21, 2005
More Exchange Troubleshooting
Today, I was the victim. Yesterday, my wife told me that she had sent me an e-mail. I didn't get it. I didn't think much of it at the time, because we were getting ready to build yet another fence in the yard. this morning, however, she let me know that she got several "mail delivery delay notifications" including one for the mail she sent me. Here's how the troubleshooting process went.
- I took a look at her mail delivery delay notification message. It came from my Exchange server. Not good.
- I opened Exchange System Manager on my server, expanded Servers, my server name, and clicked on Queues.
- I noticed that the queue going to my smarthost is in a Retry state and had 25 messages in it. I clicked on the mail queue, and in the Additional Queue Information window at the bottom of ESM was a notice about the mail server refusing authentication. Great.
- About two weeks ago, my DHCP IP address from my ISP changed. First time in 2.5 years. I thought, hmmm, maybe they're no longer requiring authentication on the outbound mail server. So I went through the process to turn off authentication on outbound SMTP connections. I had to turn this off both in the SMTP Connector and the SMTP Virtual Server. Then I restarted the SMTP service.
- All the mail that was in the queue bounced back undeliverable - needs authentication. Crud. Now my wife will have to resend all that mail. Oh, and all the mail I sent out over the weekend will have to be resent, too.
- I went online to change my password with my ISP. After spending 30 minutes trying to navigate their site to get to a point where I could change the password (no, it's not obvious to the casual observer), I went back through and reconfigured the SMTP Virtual Server and SMTP Connector to relay mail through a mail host. I then restarted the SMTP service and sent a test message.
- The test message sat in the outbound mail queue, which was in a retry state. Fine.
- I opened Netmon and ran a trace while forcing a connection attempt to my ISP's mailhost. Here's a heavily-edited transcript of that session:
220 mailserver.isp.net -- Server ESMTP (Sun Java System Messaging Server 6.2
EHLO myservername
250 mailserver.isp.net
250-8BITMIME
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-AUTH PLAIN LOGIN
250-AUTH=LOGIN
AUTH LOGIN Base64EncodedUserName
501 5.5.0 Invalid input (Invalid authentication protocol).
[connection dropped]
- I looked up the RFC specs on the SMTP AUTH protocols and found that this was a completely valid mechanism for submitting authentication. Hmmm.
- I set up Outlook Express with the same authentication information and ran another Netmon trace. Here's the heavily-edited results:
220 mailserver.isp.net -- Server ESMTP (Sun Java System Messaging Server 6.2
EHLO myservername
250 mailserver.isp.net
250-8BITMIME
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-AUTH PLAIN LOGIN
250-AUTH=LOGIN
AUTH LOGIN
334 [Base64encoded - username prompt]
[Base64encoded - username response]
334 [Base64encoded - password prompt]
[Base64encoded - password response]
235 2.7.0 LOGIN authentication successful.
- I noticed that there's a key difference between the two sessions. The connection from the Exchange server *includes* the Base64encoded username at the end of the AUTH LOGIN command, the connection from Outlook Express does not.
- I did some Google searching and came up with the following:
http://groups-beta.google.com/group/microsoft.public.exchange2000.connectivity/browse_thread/thread/c4526021a9fd1133/eaecbd6ef6c8c514?q=exchange+2003+smtp+not+conforming+RFC%3F&_done=%2Fgroups%3Fq%3Dexchange+2003+smtp+not+conforming+RFC%3F%26&_doneTitle=Back+to+Search&&d#eaecbd6ef6c8c514
It seems that my ISP has made some settings changes that are now not accepting the Exchange version of the AUTH LOGIN command which includes the initial-response argument with it. Grrr.
- I went back into both the SMTP Virtual Server and the SMTP Connector and set them both for mail delivery through DNS, bypassing my ISP's mailhost altogether. The mail got delivered.
- I spent two hours on the phone with my ISP trying to find out what happened. My assumption was that they changed settings over the weekend that prevent outbound mail delivery through their mail servers unless the account-holder does some sort of upgrade. All four of the people I speak with, including two supervisors, have no clue what I'm talking about. I expected that when I spoke with technicians who worked in the business-class DSL technical support group that I might find someone who had at least heard of Exchange. None had, and Exchange did not show up on their list of "supported" products. I tried multiple times to get them to put me through to someone who could tell me what changes were made on their servers over the weekend. One person actually at least pretended to look up something (in her defense, she very well could have looked something up in her systems, but the result is the same) and then tell me that she has no record of any changes made to their mail servers over the weekend. Even when I threatened to pull my service from them and go with another provider, I got little more than a "well, that's unfortunate" from them.
- I posted a message (now that I have some semblance of outbound e-mail working again) to one of the mailing lists I participate in describing my problem. I got a very quick response that one of the other members has posted about this problem in her blog this morning. I did a quick read of the blog post, and sure enough, it's exactly what I'm experiencing. Apparently, the ISP *has* decided to prevent mail relaying through their systems unless you upgrade to a static IP address with them.
- I confirmed again with my ISP that I cannot get a static IP address in my area. Lovely.
- I checked with one of the vendors that the blog post mentioned as a solution for outbound mail relay service issues - http://www.dyndns.org/services/mailhop/outbound.html. For $14.95/year, I can get 150 outbound e-mails per day routed through their mail host. I signed up immediately.
- I reconfigured my SMTP Virtual Server and SMTP connector to route all outbound mail through this new service, which also includes support for encrypted SMTP connections between my mail server and their mail server, ensuring that my authentication information is not sent across the internet in clear text, something that my ISP, even if they would allow me to route mail through them any more, does not provide.
- Mail started flowing normally again, and I found no significant delays in delivery of mail from my server to recipients.
All in a day's troubleshooting work...
Posted by Q at February 21, 2005 01:01 PM
Comments
hey, are you able to send e-mails now to verizion customers. I tried the dyndns service, and all emails are going through except the ones to verizon.
thnx
Posted by: serge at February 23, 2005 07:59 AM
We use AuthSMTP - http://www.authsmtp.com - to relay all our email from our dynamic address. We chose them because their quotas are monthly not daily and they virus scan the outgoing mail.
Hope it helps.
Posted by: Nigel Riley at March 2, 2005 04:38 AM
How did you configure Exchange to use SSL for the connector?
Posted by: RQ at March 2, 2005 07:24 AM
We tested a few SMTP services - went with Authsmtp.com - http://www.authsmtp.com - mainly because they offer monthly (not daily) quotas, were more secure and virus scan the outgoing mail.
Posted by: Andrew at March 4, 2005 04:49 AM
Sorry, it's been a while since I've been through the site approving comments. I took a look at AUTHSMTP for my client, but they didn't meet his need in the end.
As to how I configured the Exchange SMTP connector to use SSL, in teh Properties of the Mail Connector, under the Advanced tab is a button fo outbound Security. there is a checkbox for TLS encryption on that page that must be enabled. Do not enable TLS encryption for any other connectors or the SMTP virtual server.
Posted by: Q at May 4, 2005 11:57 AM