Lessons Learned

Looks like there might be an issue with installing KB948110 via Automatic Updates or Microsoft Updates if you have Sharepoint on the server. I’m tracking this down at a client site, but have heard of several other instances this morning. The behavior is this:

• After installing KB948110, Sharepoint/Companyweb is not available. The message “Cannot connect to the configuration database. For tips on troubleshooting this error, search for article 823287 in the Microsoft Knowledge Base at http://support.microsoft.com.” appears in the browser when accessing the site.
• The Application Log has numerous Sharepoint errors: #50070: Unable to connect to the database STS_Config on SERVER\SharePoint. Check the database connection information and make sure that the database server is running.
• The ERRORLOG file in C:\Program Files\Microsoft SQL Server\MSSQL$SHAREPOINT\log contains the following at the end of the log: Database ‘master’ has invalid schema.

If you go into services.msc, you will see that MSSQL$SHAREPOINT is set to Automatic but not started. If you start the service, it will appear to start, but on a refresh it will show as stopped again. Attempts to uninstall KB948110 may not show the Sharepoint instance in the list. A successful uninstall of 948110 may not restore operation to Sharepoint, either.

I’m working with Microsoft on this and will update this post as new information becomes available.

UPDATE: 1:45pm
One of the factors leading to the issue has been identified. The 948110 update is not correctly identifying the Service Pack level on some MSDE instances. In cases where MSDE 2000 is at SP3, the 948110 update should not be installing, yet it is. This was the cause of the problem on the system I was working with. Other factors are involved as well, and those are still being investigated. More info as it becomes available.

UPDATE: 4:00pm
The SBS CSS support team is now officially recommending that you hold off on installing this update on SBS servers, per their blog post: http://blogs.technet.com/sbs/archive/2008/07/09/hold-off-on-installing-hotfix-948110-on-sbs-2003-servers.aspx. I’m taking the stance that I will not be installing this update on any servers with Sharepoint until another update is released.

UPDATE: 7/10/08 7:00am
OK, a few other items have been identified as causes for this issue. I’ve already mentioned the Sharepoint database being on WMSDE 2000 SP3 instead of WMSDE 2000 SP4. Turns out there are also cases where Sharepoint is running on MSDE 2000 instead of WMSDE 2000, and that can cause problems as well. Not sure how Sharepoint is getting installed on MSDE 2000 instead of WMSDE 2000, as with the SBS 2003 install it goes on WMSDE for sure (and I think the default install of WSS 2.0 does as well), but there have been some instances where this is the case.

If you look in the ERRORLOG file in the path mentioned earlier, you may see something like this at the top of the file:

Microsoft SQL Server 2000 - 8.00.2039 (Intel X86)
May 3 2005 23:18:38
Copyright (c) 1988-2003 Microsoft Corporation
Desktop Engine on Windows NT 5.2 (Build 3790: Service Pack 2)

The last line above is the tell-tale indicator of which version of SQL that the Sharepoint database uses. If it says “Desktop Engine” like in the example above, Sharepoint is sitting on MSDE (which has a 2GB file size limit and the real reason it wants to sit on WMSDE). Instead, the line should read “Desktop Engine (Windows)” which indicates that it’s sitting on WMSDE.

Also, the SBS Blog has an update on how to get Companyweb working again if you hit this scenario. this is a workaround, as their advise is to roll back the BINN directory under MSSQL$SHAREPOINT to the content it had before the update. This can be done by restoring from backup, or by using the Previous Versions feature if VSS has been enabled on the volume. Regardless, if you have NOT installed this update yet, DO NOT install it yet. This update has been pulled out of our process for installing updates on our managed servers until the installer gets fixed.

Still, if your Sharepoint database instance has not been updated to WMSDE 2000 SP4, you should probably look to do that at you earliest convenience.

Ran across an unusual one this week that’s worth sharing. A site had two users who could not log in to Outlook Web Access hosted on SBS 2003. All other users could log in to OWA without issue, but these two could not. The employees do shift work and sign on to a shared workstation and only access e-mail via OWA, no Outlook client was installed on the workstation. The error encountered when trying to log in was “username or password is incorrect.” The password for the accounts were changed, and the accounts were checked to make sure they were not locked out. Attempts to access OWA from any workstation failed, internally and externally.

We checked the status of the mailbox in Exchange System Manager to make sure the mailbox had not been disconnected on either account, and the mailboxes were connecting fine. We tried to access the mailbox by creating an Outlook profile on another workstation and could access the contents of the mailbox, so we knew the mailbox was not corrupt. We tried to access the user mailbox through the Administrator’s OWA logon (after granting the Administrator account full access to the user mailbox) and as soon as we attempted to open the path to the user’s mailbox, we got a login prompt instead of access to the mailbox.

We tried to access the mailbox via Outlook Mobile Access, and got an “access denied” error after three login attempts. That prompted us to go look in the Security Log on the server, and that’s where we found the clue - we got a login failure for the user on the server. We found out that the local administrator had tried to restrict the user’s ability to log in to only one workstation in their AD account properties. In the Account tab, in the Log On To button, the only machine listed was the workstation. We added the server to the list of machines the user could log into, and we were able to access the account through OWA from all workstations.

Trying to restrict the user’s ability to log in to a single workstation is a good idea. But the actual authentication for OWA/OMA actually takes place on the server, which is where the service runs to grant access to the user. If you choose to use the Log On To feature of Active Directory to limit where the user can log in, be sure to add the server as one of those machines so network services can be accessed by the user account.

Microsoft released KB948496 which is an update that disables ALL of the Scalable Networking components that were added into Windows Server 2003 SP2 last year. The previous update only disabled two of the four components, and in practice, systems have continued to have problems when any of the Scalable Networking components were enabled.

This update could come down with Automatic Updates this month, but may not get automatically installed. If you are running SBS 2003 with Windows Server 2003 SP2, you need to install this update.

I ran into an interesting one today that I had not seen before. A client installed ISA 2004 on his SBS 2003 server, and we followed the best practices for doing so. After an hour or so, he called me back because he could no longer check e-mail with Outlook. I had assumed (incorrectly, of course) that when he mentioned still using POP3 to get e-mail because he hasn’t switched over to SMTP delivery yet, that he was referring to the POP3 Connector in SBS. In fact, he was still having the workstations pull down e-mail from the external server using a POP3 account in Outlook, then saving the new mail into the Exchange profile. And Outlook could not connect to the POP3 server.

We had already installed the firewall client, so I knew it wasn’t an issue with not having the client installed. I ran a monitoring scan in ISA, and saw the connections from the workstation getting denied by the SBS Internet Access rule. I checked that the Internet Users security group got created during the ISA installation, and I checked that all the users had been added to the Internet Users security group. I checked that the SBS Internet Access rule was built as it was supposed to be. All these things checked out.

I connected to the workstation and ran a manual telnet to port 110 on the POP server expecting the connection to be refused. It wasn’t. It worked as expected.

Google to the rescue again. I found this article on isaserver.org that pointed out the default configuration of the ISA firewall client in ISA 2004 is to ignore connections from outlook.exe. When this happens, ISA will treat connections from the workstation as a SecureNAT client when the connection comes from Outlook, and that is specifically denied by the SBS rules.

The workaround in the article is to change the default settings for the firewall client in the ISA Management Console so that the Firewall Client will take connections from outlook.exe and pass them through ISA as a firewall client and not a SecureNAT client, and this change allowed the workstation to pull e-mail down from the remote mail server as it had before ISA was installed.

Long term, the my client will be moving to direct SMTP delivery of e-mail. Near term, he will be configuring the POP3 connector to pull mail into Exchange instead. But it was the first time I’d worked with a setup where Outlook on the client was pulling e-mail from a remote POP mail server behind an ISA server, and it caught me by surprise. Hopefully this post will help someone else in this situation find the solution a little quicker.

There are a lot of changes happening in the backup industry as the space begins to move away from tape as the primary backup medium and starts using hard disks or network storage instead. Several vendors are now offering backup tools that rely on imaging technologies instead of file-based backups. I have started migrating many of my clients over to image-based backup tools, in fact.

But there’s still one thing that you really, really need to do when working with image backups - System State Backup. This is a special backup process that backs up Active Directory and other key server information such as the registry and other Windows configuration settings. I can’t count the times I’ve run across a situation that would have been easily resolved by restoring a system state backup. AD corruption, GPO corruption, etc. Sure, you could restore the entire C: image with your imaging tool, but then you lose any other data that was added to the drive following the backup.

But there are also some cases where an image-based backup fails to do its job. I spoke briefly with someone today who was having trouble because the image-based backup tool he was using was not correctly restoring the data to the system partition and the system was not bootable. He had gone around and around with the vendor of the backup software, and they could not get it to work. My first question to him was “do you have a system state backup?” Unfortunately, no. If he’d had a system state backup, he could have done a core install of the server OS, restored the system state, then gone into the backup software and done a file-based restore of the remaining contents of the system partition.

A system state backup can be captured very easily from ntbackup on a server, and can be saved to a file on local disk or on a share to another machine on the network. Either way, the backup file should be stored someplace that it can be easily accessed in case a restore is needed.

My operation manages security updates for a number of clients running SBS. This is a process we handle remotely, and have the process down to nearly a science. Every once in a while, we encounter hiccups, but not very often. This weekend, we found several servers that got “stuck” in a state following a restart request where the server was still up, but it wasn’t responding to RDP requests.

This behavior has been noted by several folks in the community, but it’s been a hit and miss prospect to figure out what’s going on. Well, at the time you’re trying to get updates installed for a client, you’re not really all that concerned about the “why” of it all. You just really want to get the server back to a point where you can connect in to it again without having to go onsite. And given that we manage servers all across the US, going on site just isn’t an option.

Some folks have taken to using third party remote control tools to access their servers rather than relying just on RDP. Still it’s possible that these services, like the TS service, get stopped when the server restart command is issued and a remote connection still isn’t possible.

Fortunately, with SBS, we still have an option available to us to help get the server restarted so we can get back in: Remote Web Workplace. In all of the cases we encountered this weekend, it was only the TS service that got shut off, so we were able to log in to RWW, connect to a workstation at the site, and get the server restarted from there.

But wait, that’s the real magic of this post - how to remotely restart the server when you cannot connect to it by other methods, but it’s still alive on the network. Here’s how:

1. Log in to the workstation via RWW as the domain administrator.
2. Verify that the server is actually “alive” by connecting to the server with the Computer Management console:
1. Right-click on My Computer on the workstation and select Manage.
2. Right-click on Computer Managemen (Local) and select “Connect to another computer.”
3. Enter the name of the server and click OK.
4. If the connection succeeds and you can browse the event logs on the server, you’ve got a good connection.
5. From within the Computer Management console, you may be able to restart the service that got stopped, in this case the Terminal Server service. expand Services and Applications and click on Services to see the list of services. Find the service in question and see if you can start it. This may still not get you what you want, so you may need to proceed with the steps to restart the server.
3. Open a command prompt on the workstation.
4. Type “shutdown -r -m \\servername -t 5″ (without the quotes) and press Enter. This will restart the server servername after a 5 second delay.
5. When you get kicked out of the RWW session to the workstation, you know the server has finally restarted.

There are lots of things you can do with the shutdown command. Type “shutdown /?” to see what the various options are.

If you encounter this problem and do NOT have an SBS server (and therefore no RWW to access another workstation), you could make a VPN connection to the network and remotely control another workstation from there. The key thing is to make sure that you are authenticated as the domain administrator when you issue the shutdown command or you’ll get access denied errors and still won’t be able to do anything. Or if you have remote access into a workstation on the network using some other means, the same shutdown option will still work.

I keep managing to run into this scenario at just about every site I’ve worked with in the past 6 months, and there’s just enough time that passes between each setup that I can be prone to forgetting the issues I ran into on the previous pass. Today was no exception.

I was setting up a workstation for a client and there were two profiles that they needed “kept” as part of the ConnectComputer wizard process. No problem, handles that like a dream. Except both came back with the dreaded “The following user settings are private” error. There is a Microsoft KB article ( KB886210) on this error, but buried at the bottom is the real meat of the resolution - how to find the actual file/folder that’s tripping up the wizard.

Fortunately, you can look in the SBSNetSetup.log file (stored by default in the C:\Program Files\Microsoft Windows Small Business Server\Clients folder on the workstation) and scroll down to the end of the file to find the file or folder that’s tripping up the wizard. Usually (I recall this now, mind you) it’s a file somewhere in the Temporary Internet Files that can be deleted to fix the issue, but the log file will tell you specifically which file it is.

What the KB doesn’t mention, however, is that there may be more than one file that will interrupt the wizard, and you won’t know what it is until you hit the next run of the wizard. Don’t panic, though, as you simply check the SBSNetSetup.log file after each failure of the wizard to find the next file that is causing problems. Once you clear all the files, the wizard will complete as expected.

I wish I knew what was causing this to happen (at least in my world) much more frequently of late, but now that I remember the issue enough to blog it, it should cause less constarnation in the future.

Went through the wringer on this one and could not find any clear documentation/resolution on it on the web, so after discovering the cause/fix, I thought it’d be a good idea to post. I had just finished a Swing Migration for a client when the client reported problems with OMA. The core problem was a certificate error on his Windows Mobile device that wouldn’t allow him to sync, but since he knew about OMA, he tried to use that instead, but got errors. Specifically, he got the following error after logging in to the OMA interface:

A System error has occurred while processing your request. Please try again. If the problem persists, contact your administrator.

Outlook Web Access worked fine on the server, as did the full Outlook client. It was just OMA that was having fits.

I got in and looked at the event logs on the server (after verifying that other accounts got the same error with OMA) and found an MSExchangeOMA 1503 error. The full text of the message is really verbose, so I won’t paste it all here. I dug around Google and eventid.net, but didn’t really find anything that pointed to a fix.

Finally contacted another resource who pointed me to the solution. Turns out that at some point during the migration, the homeMTA attribute for the user accounts got munged. The homeMTA attribute value looked similar to this:

CN=Microsoft MTA\0ADEL:111e6f10-7865-41da-8c30-8d249bf3a050,CN=Deleted Objects,CN=Configuration,DC=domain,DC=local

Well, the Deleted Objects container was a clue that it wasn’t pointing to the correct place. I created a new test user on the server and looked at the homeMTA attribute value for that user, which was:

CN=Microsoft MTA,CN=LEONSERVER,CN=Servers,CN=first administrative group,CN=Administrative Groups,CN=LS,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=local

After changing the homeMTA attribute on the Administrator account, OMA worked fine. All user objects that were present prior to the migration had incorrect values for homeMTA, and I adjusted them all. Case closed, OMA working.

Where do you find the homeMTA attribute? In ADSIEDIT. ADSIEDIT is not for the faint of heart. It’s actually more invovled that registry diving. But if you encounter this error, the fix is relatively straightforward.

First, you need to install the Support Tools package on the server. In an SBS install, the Support Tools installer is on CD #2 in the \SUPPOR\TOOLS folder, and the installer is named SUPTOOLS.MSI. Once you install that, you can access ADSIEDIT by running adsiedit.msc from a command prompt. Expand the Domain node, and browse into DC=domain,DC=local -> OU=MyBusiness -> OU=Users -> OU=SBSUsers. Right-click on a user object and select Properties. Then scroll down the list of attributes until you find the hostMTA entry. Find the user that will have the correct value and copy the value, then paste it into the user that has problems. Apply the changes and OMA should immediately work.

Ran across one today that hasn’t been documented to death in the ether, so it’s worth sharing. Bottom line, if you install Windows Sever 2003 SP2 and have ISA on the box, you dang well better follow KB927695 and disable Receive Side Scaling on your NICs. even better, don’t do the hack in the registry, just modify the properties of the NIC to disable the setting. Here’s another reason why:

I was working with someone who was having a number of problems on his new SBS box. We got a number of them fixed, then we were trying to join a workstation to the domain. He had tried many variations of this previously, but all had failed. Once we resolved his IIS issues, we decided to see what would happen with the ConnectComputer wizard.

Problem 1: You can’t get to the ConnectComputer wizard. We continually got a “page cannot be found” error when trying to connect to http://server/conectcomputer, just like the Add Client Workstation wizard says to do. We could access the page at http://internalIP/connectcomputer, but this doesn’t always work, either. We were finally able to get the page to load and at least get through the main portion of the wizard using https://server/connectcomputer/

Problem 2: The ConnectComputer wizard encountered errors and could not complete. This happened at the end of the wizard as it was trying to change network settings to initiate the reboot that would join the machine to the domain, etc., etc., etc. We looked in the client-side log for the ConnectComputer wizard (which is in C:\Program Files\Microsoft Windows Small Business Server\Clients\SBSNetSetup.log, by the way) and found the following error in the log:

NetJoinDomain() failed [1727]

Google found only a few posts about this specific error, mostly having to do with trying to join a workstation over a VPN when ISA is involved. Well, this server had ISA installed, but this is a local workstation and not over a VPN. Also worth noting is that it’s the first workstation to join the domain. But I digress. We followed the advise about turning off Strict RPC checking in ISA (which I regularly forget to do and hate that I have to in the first place) but that had no effect. Just when I was about to punt, I discovered that SP2 had been installed on the box.

Yes, the dreaded Windows 2003 Server SP2. The one that has actually been causing more issues than MS cares to admit right now. And the only reason he installed it when he built the server? Because it was listed in Microsoft Updates through the web, and since it’s up there, it must be safe to install, right? In this case, I certainly wouldn’t have installed it, but that’s just me. Oops, digressing again.

So I reviewed the Official SBS Blog for stuff about SP2 and found the note on the Receive Side Scaling. The server had broadcomm NICs (which have issues in themselves), so I went into the NIC settings through Network Connection Properties and disabled Receive Side Scaling on both the internal and external NIC. Viola! ConnectComputer not only ran successfully, but we were able to access it through http://server/connectcomputer without SSL.

I’ll be darned if I can understand exactly why changing this setting when SP2 and ISA are on the box had this type of impact on local networking, but as soon as I changed it, everything worked. I liken this to the other bizarre resolution where changing the internal name on a security group allows the Connect to the Internet wizard to run correctly (look down at the last entry in the thread for the real resolution) - can’t explain fully why it works, but it does.

Moral of the story - read everything about SP2 on the SBS blog and even if you think you may not be affected, look at each one of the items listed there. Or don’t put SP2 on any of your boxes just yet. The latter is the direction I’m taking when I have an option.

A couple of days ago, I ran across a couple of very handy little tools that I had previously been unaware of: the IIS Diagnostics Toolkit, and the SSL Diagnostics Toolkit. Technically, the SSL Diagnostics Toolkit (ssldiag) is part of the IIS Diagnostics Toolkit (iisdiag), but it’s actually been updated since iisdiag was released, and if you’re trying to troubleshoot SSL issues, you’ll want to download and install it separately. The tools are small and straightforward to use, and they’re free!
I ran across these tools trying to troubleshoot an OWA problem - after a rebuild on an Exchange box, OWA and Exchange ActiveSync would not work. A bit of digging revealed that IIS didn’t seem to be responding to any HTTPS requests at all. After digging around a bit and finding these tools, I ran a quick SSLDiag check, and it immediately told me that the SSL certificate that had been installed in IIS did not have a private key associated with it, which meant that IIS could not generate any encrypted information to send back to the requesting client. We installed a new cert, and the site started working as expected.

So now I have another set of tools that I’m going to include as defaults to install on a new server build in addition to the Support Tools from the server install CDs. Very handy tools to have.