Archive for How To
Automating the WSUS 3.0 Cleanup Process
Posted by: | CommentsWhile I’ve not been a huge fan of WSUS in the past, it’s been growing on me over the last year or so. Specifically, I’ve been really pleased with how WSUS 3.0 and SBS have been integrated (well, so long as you don’t hit a problem with the integration, which can then lead to a LOT of work to recover or repair or reinstall, but that’s a different post for a different day). But there are still challenges to keeping WSUS in check and keeping it from having unintended impacts on those same SBS servers.
Fortunately, most of the commonly-encountered problems with WSUS 3.x can be dealt with by running the Server Cleanup Wizard from the Update Services console. [NOTE: If you have never run the Server Cleanup Wizard in WSUS on a server that's been in production for a while, I recommend running the wizard manually and only select one category at a time. The first run can clean a LOT of information out of the WSUS environment, and it can take a VERY long time to complete.] But in this day of automating tasks, I don’t want to manually run the Server Cleanup Wizard on a regular basis as it can still take some time to complete the supplemental runs even after the first (and potentially longest) pass has been completed.
Well, there are two mechanisms for automating the Server Cleanup Wizard process on an SBS 2008 server (and other servers running WSUS for that matter). The first method that I’ll discuss below is fairly easy to google, but the second doesn’t show up in searches related to SBS 2008 (that I’ve been able to find at the time that I put this post together), so I’m going to document it here.
Let me start by saying that a lot of people who have implemented one of these two methods seem to be in agreement that these processes (or a variation thereof) should be included within WSUS itself and not relegated to what amounts to an add-on for maintenance and management. I’m in the same category, and really would like the WSUS team to look at providing tools with WSUS to be able to schedule the maintenance out of the box.
The first solution I ran across last year was a tool uploaded to Codeplex: http://wsus.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=17612. This is a complied tool that will perform operations on the WSUS implementation based on command-line parameters that are passed to the tool when executed. It can run each of the cleanup tasks in the Server Cleanup Wizard individually or in groups, and also includes an SQL script that the tool can call to perform maintenance on the WSUS database file itself. I’ve deployed this in testing on a few SBS 2003 installations where I have WSUS 3.x running, and it’s been able to keep the WSUS installation in check rather nicely. My only beef with the tool is that since its a compiled executable, it’s impossible to tweak its operation beyond what the developer has coded into the tool. Currently, I can’t think of any WSUS tasks that I’d like to do that this tool cannot, but if an update to WSUS changes the way some of these tasks can be called, it’s possible that the tool might cease to function or not be able to handle new functionality and need an update from the author. I’ve also not run this on SBS 2008 yet simply because I don’t have a test box that I could run this on to make sure it doesn’t misbehave on that platform. It might work just the same on SBS 2008 as SBS 2003, but I can’t confirm that first-hand, so I haven’t pushed in out.
The second solution I ran across (again, not in an SBS 2008 search) is a PowerShell script that calls the Server Cleanup Wizard functions from WSUS directly. Since PowerShell is enabled by default on SBS 2008 out of the box, and since I can get into the code directly, I went ahead and implemented this script on my own production server, because I honestly hadn’t run the Cleanup Wizard on it in I don’t know how long. The script came from the Microsoft Technet Script Center at http://gallery.technet.microsoft.com/ScriptCenter/en-us/fd39c7d4-05bb-4c2d-8a99-f92ca8d08218. I have a Tools folder on the root of the second partition of every server I deploy, and I added a Scripts folder in that to house this script. I named the script WSUS_Cleanup.ps1 and copied the contents from the Script Center page into the file. I then opened a Command Prompt as Adminstrator and ran “powershell.exe WSUS_Cleanup.ps1″ on the server. After a long wait (like I said, I hadn’t run the Server Cleanup Wizard in a looooooong time), I got output from the script that showed the results of each of the steps it ran within the script (as listed on the Script Center page, the option to remove old computer from WSUS has been commented out).
Being the kind of guy who likes to review the results of processes once they complete, I build a quick and dirty batch file wrapper for the PowerShell script. Yes, I probably could have done the whole thing in PowerShell, but I’m still a bit of a PS newbie, so I relied on my comfort with batch files to get this wrapper done. Here’s the contents of the WSUS_Cleanup.bat that I put on the server:
@echo off @echo Starting cleanup: %date% %time% >> d:\tools\scripts\WSUS_Cleanup.log powershell.exe d:\tools\scripts\WSUS_Cleanup.ps1 >> d:\tools\scripts\WSUS_Cleanup.log @echo Finished cleanup: %date% %time% >> d:\tools\scripts\WSUS_Cleanup.log
The batch file writes the current date and time to a log file that I created in the same Scripts folder where the other pieces are, then calls PowerShell to run the cleanup script and appends the output of that process to the log file as well. Once that finishes, the current date and time are again appended to the log. Now I can see when the script ran, what it did when it ran, and how long it took to complete.
Either of these tools are easily adaptable to running as scheduled tasks or as scripts from your favorite RMM tool. THE WSUS_Cleanup from Codeplex has a couple of advantages over the PowerShell script. One, you can select which components of the Cleanup Wizard you wish to run by adjusting the command line call to the tool. With the PowerShell script as written, you have to modify the script and comment or uncomment each of the tasks. (Yes, a savvy PowerShell person should be able to modify that script to mimic the behavior of the Codeplex tool, and as I’ve mentioned, I’m not that guy. Yet.) Second, the Codeplex tool has the SQL maintenance script included which can be run within the scope of the Codeplex tool. The PowerShell script does not include anything for SQL maintenance on the actual database files. Again, someone with SQL skills could easily script up and automate a process to do the same thing, and again that’s not me.
Given that PowerShell is getting more and more visibility in the Server 2008 world, I’m going to be focusing (when possible) on dealing with automation tasks that make use of PowerShell or other native scripting tools rather than rely on someone else to build an executable file. Not to say that the WSUS_Cleanup tool on Codeplex is a bad thing. I’m probably going to keep that on my 2003-based systems until there’s a reason not to. But for my 2008 deployments, I’m going to stick with PowerShell for WSUS maintenance. If nothing else, I get an excuse to learn more about PowerShell and keep my WSUS installations in good working order.
Recovering “Hidden” Disk Space Used on SBS 2008 C: partitions
Posted by: | CommentsOne of the significant differences in the minimum specs for installing SBS 2008 versus SBS 2003 was the minimum size of the C: partition needed for installation and operation. SBS 2008 requires a minimum of 60GB in the install partition or it won’t go. Those of us who were used to fighting the 12GB C: partition implemented by OEM vendors in SBS 2003 initially looked at that and thought “yeah, that’s a good change.” Well, as it turns out, kinda like the 4GB RAM minimum spec, the 60GB C: partition may not be big enough after all.
If you ask around those who have been doing SBS 2008 deployments, one of the best practices adopted by most is to use the Move Data Wizards in the Server Storage tab of the SBS 2008 Console and get the key data components off the C: partition and onto another partition (Exchange, SharePoint, User’s folders, User’s redirected documents, and WSUS content). And if you take the step that some do of installing third-party software to a partition other than C:, we should be ending up with a fairly pristine C: partition with minimal dynamic data on it. In theory.
I’ve been deploying my SBS 2008 installs with a 100GB C: partition simply because I figured that over time, something would find a way to suck up all the space on C: and we’d eventually get to a point where we’d have to deal with resizing paritions or doing manual data cleanup. I didn’t expect that I’d hit that scenario just over a year after my first SBS 2008 production deployment.
In the last couple of weeks, my monitoring tools have started chirping about low disk space on C: on a couple of installs. Sure enough, one installation had 17GB remaining of a 100GB partition, another had 3.5GB remaining on an 80GB partition (my own production box, and yeah, it really needs an overhaul, but that’s another story). I started digging around and found the most common disk hog that’s been complained about across the net, the winsxs folder. Based on everything I’ve been able to read about winsxs, including a post from the Windows Server Core Team, that’s something that we’ll just have to live with, and really isn’t the point of this post anyway. Still, on my boxes, the winsxs folder still only amounted to about 12GB (bigger than what I’d like, but certainly not the primary culprit) which is only about 10% of my standard install C: space. Something else had been sucking away space and keeping it from me.
We use TreeSize from JAM Software as a standard utility on our server deployments to help monitor disk space usage, as this is something that comes up from time to time. [NOTE: this is not a specific endorsement of TreeSize, just a note that it's one of the many tools that we use in our operation.] So in the case of these low-free-space servers, I fired up TreeSize and went looking for the disk hog. Surprisingly, I couldn’t find it. I did clear up some areas that showed a larger-than-expected usage, but couldn’t find the smoking gun. A few weeks have gone by, and while I’ve been monitoring the state of these servers to ensure that free space didn’t get critically low, other tasks moved up on the priority list.
Then a discussion on one of my private lists cropped up regarding this exact topic, and I learned two valuable tidbits from that discussion.
The first is that in order for TreeSize to see the contents of ALL folders on the C: partition, it must be Run As Administrator. Upon reflection, this makes sense, but I know it’s catching a lot of experienced system admins off-guard. Some are advocating disabling UAC on the server to avoid this kind of issue, and I’m honestly not fully decided where I stand on that, so I won’t comment either way on that. But it does serve as a reminder that many system tools we may have been using for years on 2003 servers might not behave the same way under 2008 if you don’t use the almighty Run As Admin option.
The second is that the WSUS site in IIS has been logging an OBSCENE amount of data into the IIS logs folder. One of my servers had nearly 30GB (yes, that’s 30 gigabytes) of data in the WSUS log folder (C:\inetpub\logs\LogFiles\W3SVC1372222313). Another had just over 20GB. And in looking in the folder, I saw numerous DAILY log files that were well over 100MB each, with some well over 200MB each.
Once I cleared out the old log files (honestly, how far back am I going to need to look at WSUS logs anyway?) the free space on C: increased to a reasonable level, and my monitoring stopped yelling at me quite so often.
There are multiple lessons learned from this experience for me. The first is the whole reminder about Run As Administrator in the Server 2008 era. I’ve even taken to labeling some shortcuts with “Run As Administrator” in the icon name just to serve as a reminder. The second lesson is that 60GB is certainly NOT going to be sufficient as a minimum partition size on a production SBS 2008 server, even if all other data is moved off to different volumes (and I haven’t even covered the option of moving the WSUS SQL database files off of C: to another partition, which can’t be done through wizards but must be done by hand). With winsxs and the WSUS logs as two items that will definitely be grabbing disk space unexpectedly (well, it’s expected now anyway), we can be sure that over time there will be others. And as stated on the Core Team blog, you can only expect that winsxs will continue to grow over time. If it’s 12GB now, how large will it be in a couple of years? The third lesson is that some logging that happens automatically on the server probably should not just be left unchecked. If you enable SMTP logging (which I do and recommend for troubleshooting purposes), you should clean out old SMTP logs on a regular basis. Well, now you can add WSUS/IIS logs to that approach as well. There are numerous posts out there for ways to script this process, and I’m evaluating the approach we’re going to take within our operation to make this happen for our customer base.
If you’ve been struggling with low disk space issues on SBS 2008 C: partitions, hopefully this information will help you get a better handle on the immediate actions as well as the long term strategy that you’ll develop for your particular environment.
Another reason SBCore could shut down your server
Posted by: | CommentsEarlier this month an associate pinged me about an unusual situation. He had an SBS 2003 server that was shutting itself down periodically, claiming that it was doing so because there was another SBS server in the domain. Well, this is expected behavior if there is, in fact, another SBS server in the domain, but this particular network had only one server, the SBS sever, and not a single other server or history of another server in the network. Another unusual symptom of the behavior is that the server would remain up for a little over 24 hours before it would shut itself down because of the phantom SBS server. According to MS KB 925652 the SBS server will shut down every hour if it detects another SBS server in the domain, so clearly a different set of events were causing this behavior. The server was logging SBCore 1011 errors in the event logs, but only after the server had been online for about a day.
On a tip from a colleague at MS, we started to look for a possible memory leak in the system. I worked with my colleague to set up perfwiz and poolmon to try to identify the process (or processes) that were leaking. The theory was that a runaway leak could strip the server of valuable no-paged pool memory which could cause the SBCore check to fail and generate the errors and shutdown event. I must admit, perfwiz and poolmon never were my strong points, so even after we got some results back, the review didn’t come up with a smoking gun.
Then my associate found a tip that I’d not heard of before, even though I regularly modify settings where this tip was found. He opened the Task Manger on the server, selected the Processes tab, then opened Select Columns under the View menu. In here, he enabled the “Memory – Non-paged Pool” column and then sorted the Task Manager process list by that column. Sure enough, he not only quickly found the culprit, but also could sit and watch the Non-paged Pool count grow steadily right before his eyes. The service causing the problem? spoolsv.exe, the print spooler service.
A quick bit of Googling on his part ultimately led him to this post from Tek-Tips which helped him identify the root cause of the problem: HP Standard TCP/IP ports for printers on the sever. He changed the port types for the printers from HP Standard TCP/IP ports to Standard TCP/IP ports, and the server hasn’t shut down again since.
Turns out, there is a KB on this situation, too, MS KB 933999. And in going back and looking further, the server was logging the Srv 2019 errors in the event logs as well. Since we were sidetracked by the anomalous SBCore behavior, we did overlook the 2019 as a possible factor as well.
In the end, I learned two things from this. One, you can track non-paged pool memory usage in Task Manager (which really isn’t a *revelation* per se, just something that I wouldn’t have necessarily deliberately gone out and looked for), and two, memory leak issues can cause anomalous SBCore errors and the shutdown of an SBS server. The good news is that the server was shutting down “normally” because of the SBCore misfire instead of totally running out of non-paged pool memory and crashing, as MS KB 933999 points out can happen. Bottom line, customer happy, and tech support further educated!
Connecting to Exchange 2007 from Snow Leopard Mail Client
Posted by: | CommentsWith the release of Mac OS 10.6 (aka “Snow Leopard”), Apple has incorporated the ability to connect the Mail, iCal, and Address Book apps natively with Exchange 2007. Unfortunately, support for this is NOT available for Exchange 2003 servers. Here’s how to set up Apple Mail:
- Open the Mail app.
- From the Mail menu, select Preferences (or press Apple-; to open Preferences).
- Select the Accounts tab in the Preferences window.
- Click the Add button (the “+” button in the lower left corner of the Accounts window).
- Enter the Full Name, Email Address, and Password for the Exchange account (NOTE: the e-mail address needs to match the default e-mail address on the Exchange account. In other words, if your outbound e-mail shows your address as Jonathan.Dough@smallbizco.net, then enter it exactly that way. Entering it as jonathan.dough@smallbizco.net may cause problems) and click Continue.
- If your Exchange server is correct set up for Autodiscover, the server will be found and automatic configuration will be attempted. Also, if your mail server is using a private SSL certificate, you may see one or more prompts about not being able to identify the server. If you have concerns, contact the team responsible for your mail server. Otherwise, click Continue or Connect if you see these prompts, but understand that you may be putting some account information at risk by doing so.
- If your e-mail address does not match your login name (i.e., the address is Jonathan.Dough@smallbizco.net, but the login is jonathandough), you will be prompted to enter your login credentials. Change the username to match the username you use to sign into Outlook Web Access, then click Continue.
- Once the account setup has confirmed the connection to the server, you will be prompted to set up your Address Book contacts and iCal calendars. Uncheck these boxes if you do not want to synchronize your iCal calendars with your Exchange calendars or your Address Book contacts with your Exchange contacts. Click Create when finished.
- Close the Accounts window when complete.
Now you will see a set of mail folders for your Exchange account under the Mailboxes section of Apple Mail. It may take some time for the folders to synchronize if you have quite a bit of mail on the server.
NOTE 1: When setting up Apple Mail to communicate with your Exchange 2007 server in this way, you are working directly with the information that is on the server, NOT on a local copy that has been downloaded on your Mac. That means that if you delete an e-mail from Apple Mail, it is immediately deleted from the server and will NOT be available to any other mail clients you may be using to access the information on your Exchange server (i.e., Outlook on a PC or a mobild phone that has native Exchange connectivity).
NOTE 2: If you have done an in-place upgrade from a previous version of Mac OS 10 to 10.6, there is a change that iCal may not be able to synchronize with the Exchange server. I have run into this issue and have not yet been able to find a solution, but others have not encountered this problem, so it’s unclear what the exact cause is at this point.
Connecting iPhone 3.x to Exchange
Posted by: | CommentsI originally posted instructions for configuring an iPhone 2.0 device to an Exchange server back when the iPhone 3G was originally released. While those instructions still hold for the most part, the iPhone interface has changed somewhat, so here’s an updated post with pics for connecting the iPhone 3.x OS to an Exchange server.
- From the Home screen (unless you’ve relocated the icon) open the Settings app.
- Click the Mail, Contacts, Calendars item in the list.
- Click Add Account.
- Click Microsoft Exchange at the top of the list.
- Enter your e-mail account information.
For the Email field, enter your default e-mail address exactly as it appears on your outgoing messages. This is important: if your outgoing e-mail address is Jon.Dough@smallbizco.net, you must match it exactly. If you enter jon.dough@smallbizco.net, you may encounter issues with calendar items. You may not need to enter the Domain field, but do enter it if you know what it is. Your username and password should match what you enter to access Outlook Web Access or your network account. If you do not know this information, you will need to get it from your system administrator. - Once you have entered the account information, click Next. The iPhone will attempt to connect to your mail server. You may be prompted to enter the name of the mail server if it could not be found automatically. This is the same as the server you use to access Outlook Web Access. If your Outlook Web Access server is https://remote.smallbizco.net/owa, then you need to enter remote.smallbizco.net as the mail server. Then click Next.
- If the iPhone cannot correctly validate the security certificate on your mail server (this may be the case if your Exchange server is running on Small Business Server 2003 or 2008), you will be asked what to do about the connection. If you know you have entered the correct information about your mail server, click Accept. If not, click Cancel.
- Choose which items from the Exchange server you want to sync with your iPhone. If you already have your contacts or calendar synchronizing with another source, you may want to hold off on selecting those to avoid a potential loss of data.
- Click Done when finished.
The iPhone will now start the initial connection to the Exchange server and synchronize the selected information.
Getting your IP back
Posted by: | CommentsSo you’re having trouble getting to the Internet? Can’t ping the Internet gateway? Can’t ping your own IP address? Have network adapters that refuse to enable or disable? Could be a corrupt IP stack. You can take a look at MSKB 299357, or you can follow these steps:
- Make sure you’re logged in with a local administrator account.
- Open a command prompt.
- Run the following command :
netsh int ip reset logfile.txt
where logfile.txt is the name of a file where the command can write its output. - When the command completes, run it again with a different filename for the output file.
- When that run completes, run it one more time, again with a different filename for the log file.
- Restart the computer in Safe Mode with Networking.
This will reset the TCP/IP settings back to sane defaults, which means all adapters in the computer will be set for DHCP. If you’re doing this on an SBS server, restarting in Safe Mode with Networking is absolutely crucial in order to avoid the dreaded 30 minute reboot. When the computer comes back up, set the network settings as needed, then reboot normally.
You may still have other issues, but these steps will get you a nice, clean, DHCP-enabled set of network adapters in the system.
Removing Device Security Code from iPhone Configured for ActiveSync
Posted by: | CommentsExchange 2003 SP2 and Exchange 2007 have options to require a security code on a device that will connect to the Exchange server using ActiveSync. This setting is optional in Exchange 2003 but is enabled by default in Exchange 2007. Without getting into the reasons why you might want to reconfigure Exchange 2007 so that ActiveSync devices do not require a device security code, if you do change the Mobile Device settings after an iPhone has already connected with ActiveSync and is requiring the password, you have to jump through a couple of hoops to actually get the iPhone to pick up the new security settings.
OK, they’re really small hoops, but it’s worth pointing out nonetheless because I had to Google quite a bit to uncover this tidbit. To remove the security code requirement from the iPhone, do the following:
- Remove the Exchange account from the iPhone configuration.
- Turn off the security code in the iPhone settings.
- Add the Exchange account back to the iPhone configuration.
That’s it. If you’re prompted to create a security code when you re-add the Exchange account, then the Exchange policy hasn’t been modified correctly, and you need to dig into that. But if the requirement for the device security code has been correctly changed, you will not be prompted to enter a security code in step 3 above, and no reset of the iPhone is needed.
SSL Certificate Validation
Posted by: | CommentsI put up a post this morning regarding SSL certificate request validation over on the Third Tier web site. If you’ve been wondering how SSL certificates work in SBS 2008 or if you’re about to renew an SSL certificate on an SBS 2003 box, you might want to check out that post.
Remotely Installing This Month’s ISA Update
Posted by: | CommentsJust a heads-up for those of you who remotely install security updates for your customers. This month includes an update for ISA, and if you don’t know about it beforehand, you could end up in a bit of a jam.
As expected, when installing the ISA update, access to the Internet through the server is interrupted. Unlike some previous updates, however, when the installation of this update completes, Internet access is NOT restored. You don’t get Internet back until you restart the server.
So if you don’t have some mechanism in place for restarting the server automatically after updates install, you could find yourself, and your customer, in a rather unexpected place.
Troubleshooting Delayed Message Delivery in Exchange
Posted by: | CommentsAs more and more anti-spam solutions start doing “interesting” things with SMTP and mail delivery, there is an increased chance of users reporting that mail messages to certain domains are delayed. Unlike a full non-delivery report (NDR) which will list the SMTP error codes for easy identification of the reason for the rejection, a delayed delivery report could be the result of an Internet connection issue, spam filter, offline server, or any number of other causes. The remainder of this post details how to track down possible causes for Internet delivery issues. Read More→