Archive for January, 2010
While I’ve not been a huge fan of WSUS in the past, it’s been growing on me over the last year or so. Specifically, I’ve been really pleased with how WSUS 3.0 and SBS have been integrated (well, so long as you don’t hit a problem with the integration, which can then lead to a LOT of work to recover or repair or reinstall, but that’s a different post for a different day). But there are still challenges to keeping WSUS in check and keeping it from having unintended impacts on those same SBS servers.
Fortunately, most of the commonly-encountered problems with WSUS 3.x can be dealt with by running the Server Cleanup Wizard from the Update Services console. [NOTE: If you have never run the Server Cleanup Wizard in WSUS on a server that's been in production for a while, I recommend running the wizard manually and only select one category at a time. The first run can clean a LOT of information out of the WSUS environment, and it can take a VERY long time to complete.] But in this day of automating tasks, I don’t want to manually run the Server Cleanup Wizard on a regular basis as it can still take some time to complete the supplemental runs even after the first (and potentially longest) pass has been completed.
Well, there are two mechanisms for automating the Server Cleanup Wizard process on an SBS 2008 server (and other servers running WSUS for that matter). The first method that I’ll discuss below is fairly easy to google, but the second doesn’t show up in searches related to SBS 2008 (that I’ve been able to find at the time that I put this post together), so I’m going to document it here.
Let me start by saying that a lot of people who have implemented one of these two methods seem to be in agreement that these processes (or a variation thereof) should be included within WSUS itself and not relegated to what amounts to an add-on for maintenance and management. I’m in the same category, and really would like the WSUS team to look at providing tools with WSUS to be able to schedule the maintenance out of the box.
The first solution I ran across last year was a tool uploaded to Codeplex: http://wsus.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=17612. This is a complied tool that will perform operations on the WSUS implementation based on command-line parameters that are passed to the tool when executed. It can run each of the cleanup tasks in the Server Cleanup Wizard individually or in groups, and also includes an SQL script that the tool can call to perform maintenance on the WSUS database file itself. I’ve deployed this in testing on a few SBS 2003 installations where I have WSUS 3.x running, and it’s been able to keep the WSUS installation in check rather nicely. My only beef with the tool is that since its a compiled executable, it’s impossible to tweak its operation beyond what the developer has coded into the tool. Currently, I can’t think of any WSUS tasks that I’d like to do that this tool cannot, but if an update to WSUS changes the way some of these tasks can be called, it’s possible that the tool might cease to function or not be able to handle new functionality and need an update from the author. I’ve also not run this on SBS 2008 yet simply because I don’t have a test box that I could run this on to make sure it doesn’t misbehave on that platform. It might work just the same on SBS 2008 as SBS 2003, but I can’t confirm that first-hand, so I haven’t pushed in out.
The second solution I ran across (again, not in an SBS 2008 search) is a PowerShell script that calls the Server Cleanup Wizard functions from WSUS directly. Since PowerShell is enabled by default on SBS 2008 out of the box, and since I can get into the code directly, I went ahead and implemented this script on my own production server, because I honestly hadn’t run the Cleanup Wizard on it in I don’t know how long. The script came from the Microsoft Technet Script Center at http://gallery.technet.microsoft.com/ScriptCenter/en-us/fd39c7d4-05bb-4c2d-8a99-f92ca8d08218. I have a Tools folder on the root of the second partition of every server I deploy, and I added a Scripts folder in that to house this script. I named the script WSUS_Cleanup.ps1 and copied the contents from the Script Center page into the file. I then opened a Command Prompt as Adminstrator and ran “powershell.exe WSUS_Cleanup.ps1″ on the server. After a long wait (like I said, I hadn’t run the Server Cleanup Wizard in a looooooong time), I got output from the script that showed the results of each of the steps it ran within the script (as listed on the Script Center page, the option to remove old computer from WSUS has been commented out).
Being the kind of guy who likes to review the results of processes once they complete, I build a quick and dirty batch file wrapper for the PowerShell script. Yes, I probably could have done the whole thing in PowerShell, but I’m still a bit of a PS newbie, so I relied on my comfort with batch files to get this wrapper done. Here’s the contents of the WSUS_Cleanup.bat that I put on the server:
@echo off @echo Starting cleanup: %date% %time% >> d:\tools\scripts\WSUS_Cleanup.log powershell.exe d:\tools\scripts\WSUS_Cleanup.ps1 >> d:\tools\scripts\WSUS_Cleanup.log @echo Finished cleanup: %date% %time% >> d:\tools\scripts\WSUS_Cleanup.log
The batch file writes the current date and time to a log file that I created in the same Scripts folder where the other pieces are, then calls PowerShell to run the cleanup script and appends the output of that process to the log file as well. Once that finishes, the current date and time are again appended to the log. Now I can see when the script ran, what it did when it ran, and how long it took to complete.
Either of these tools are easily adaptable to running as scheduled tasks or as scripts from your favorite RMM tool. THE WSUS_Cleanup from Codeplex has a couple of advantages over the PowerShell script. One, you can select which components of the Cleanup Wizard you wish to run by adjusting the command line call to the tool. With the PowerShell script as written, you have to modify the script and comment or uncomment each of the tasks. (Yes, a savvy PowerShell person should be able to modify that script to mimic the behavior of the Codeplex tool, and as I’ve mentioned, I’m not that guy. Yet.) Second, the Codeplex tool has the SQL maintenance script included which can be run within the scope of the Codeplex tool. The PowerShell script does not include anything for SQL maintenance on the actual database files. Again, someone with SQL skills could easily script up and automate a process to do the same thing, and again that’s not me.
Given that PowerShell is getting more and more visibility in the Server 2008 world, I’m going to be focusing (when possible) on dealing with automation tasks that make use of PowerShell or other native scripting tools rather than rely on someone else to build an executable file. Not to say that the WSUS_Cleanup tool on Codeplex is a bad thing. I’m probably going to keep that on my 2003-based systems until there’s a reason not to. But for my 2008 deployments, I’m going to stick with PowerShell for WSUS maintenance. If nothing else, I get an excuse to learn more about PowerShell and keep my WSUS installations in good working order.
One of the significant differences in the minimum specs for installing SBS 2008 versus SBS 2003 was the minimum size of the C: partition needed for installation and operation. SBS 2008 requires a minimum of 60GB in the install partition or it won’t go. Those of us who were used to fighting the 12GB C: partition implemented by OEM vendors in SBS 2003 initially looked at that and thought “yeah, that’s a good change.” Well, as it turns out, kinda like the 4GB RAM minimum spec, the 60GB C: partition may not be big enough after all.
If you ask around those who have been doing SBS 2008 deployments, one of the best practices adopted by most is to use the Move Data Wizards in the Server Storage tab of the SBS 2008 Console and get the key data components off the C: partition and onto another partition (Exchange, SharePoint, User’s folders, User’s redirected documents, and WSUS content). And if you take the step that some do of installing third-party software to a partition other than C:, we should be ending up with a fairly pristine C: partition with minimal dynamic data on it. In theory.
I’ve been deploying my SBS 2008 installs with a 100GB C: partition simply because I figured that over time, something would find a way to suck up all the space on C: and we’d eventually get to a point where we’d have to deal with resizing paritions or doing manual data cleanup. I didn’t expect that I’d hit that scenario just over a year after my first SBS 2008 production deployment.
In the last couple of weeks, my monitoring tools have started chirping about low disk space on C: on a couple of installs. Sure enough, one installation had 17GB remaining of a 100GB partition, another had 3.5GB remaining on an 80GB partition (my own production box, and yeah, it really needs an overhaul, but that’s another story). I started digging around and found the most common disk hog that’s been complained about across the net, the winsxs folder. Based on everything I’ve been able to read about winsxs, including a post from the Windows Server Core Team, that’s something that we’ll just have to live with, and really isn’t the point of this post anyway. Still, on my boxes, the winsxs folder still only amounted to about 12GB (bigger than what I’d like, but certainly not the primary culprit) which is only about 10% of my standard install C: space. Something else had been sucking away space and keeping it from me.
We use TreeSize from JAM Software as a standard utility on our server deployments to help monitor disk space usage, as this is something that comes up from time to time. [NOTE: this is not a specific endorsement of TreeSize, just a note that it's one of the many tools that we use in our operation.] So in the case of these low-free-space servers, I fired up TreeSize and went looking for the disk hog. Surprisingly, I couldn’t find it. I did clear up some areas that showed a larger-than-expected usage, but couldn’t find the smoking gun. A few weeks have gone by, and while I’ve been monitoring the state of these servers to ensure that free space didn’t get critically low, other tasks moved up on the priority list.
Then a discussion on one of my private lists cropped up regarding this exact topic, and I learned two valuable tidbits from that discussion.
The first is that in order for TreeSize to see the contents of ALL folders on the C: partition, it must be Run As Administrator. Upon reflection, this makes sense, but I know it’s catching a lot of experienced system admins off-guard. Some are advocating disabling UAC on the server to avoid this kind of issue, and I’m honestly not fully decided where I stand on that, so I won’t comment either way on that. But it does serve as a reminder that many system tools we may have been using for years on 2003 servers might not behave the same way under 2008 if you don’t use the almighty Run As Admin option.
The second is that the WSUS site in IIS has been logging an OBSCENE amount of data into the IIS logs folder. One of my servers had nearly 30GB (yes, that’s 30 gigabytes) of data in the WSUS log folder (C:\inetpub\logs\LogFiles\W3SVC1372222313). Another had just over 20GB. And in looking in the folder, I saw numerous DAILY log files that were well over 100MB each, with some well over 200MB each.
Once I cleared out the old log files (honestly, how far back am I going to need to look at WSUS logs anyway?) the free space on C: increased to a reasonable level, and my monitoring stopped yelling at me quite so often.
There are multiple lessons learned from this experience for me. The first is the whole reminder about Run As Administrator in the Server 2008 era. I’ve even taken to labeling some shortcuts with “Run As Administrator” in the icon name just to serve as a reminder. The second lesson is that 60GB is certainly NOT going to be sufficient as a minimum partition size on a production SBS 2008 server, even if all other data is moved off to different volumes (and I haven’t even covered the option of moving the WSUS SQL database files off of C: to another partition, which can’t be done through wizards but must be done by hand). With winsxs and the WSUS logs as two items that will definitely be grabbing disk space unexpectedly (well, it’s expected now anyway), we can be sure that over time there will be others. And as stated on the Core Team blog, you can only expect that winsxs will continue to grow over time. If it’s 12GB now, how large will it be in a couple of years? The third lesson is that some logging that happens automatically on the server probably should not just be left unchecked. If you enable SMTP logging (which I do and recommend for troubleshooting purposes), you should clean out old SMTP logs on a regular basis. Well, now you can add WSUS/IIS logs to that approach as well. There are numerous posts out there for ways to script this process, and I’m evaluating the approach we’re going to take within our operation to make this happen for our customer base.
If you’ve been struggling with low disk space issues on SBS 2008 C: partitions, hopefully this information will help you get a better handle on the immediate actions as well as the long term strategy that you’ll develop for your particular environment.