Recovering “Hidden” Disk Space Used on SBS 2008 C: partitions

One of the significant differences in the minimum specs for installing SBS 2008 versus SBS 2003 was the minimum size of the C: partition needed for installation and operation. SBS 2008 requires a minimum of 60GB in the install partition or it won’t go. Those of us who were used to fighting the 12GB C: partition implemented by OEM vendors in SBS 2003 initially looked at that and thought “yeah, that’s a good change.” Well, as it turns out, kinda like the 4GB RAM minimum spec, the 60GB C: partition may not be big enough after all.

If you ask around those who have been doing SBS 2008 deployments, one of the best practices adopted by most is to use the Move Data Wizards in the Server Storage tab of the SBS 2008 Console and get the key data components off the C: partition and onto another partition (Exchange, SharePoint, User’s folders, User’s redirected documents, and WSUS content). And if you take the step that some do of installing third-party software to a partition other than C:, we should be ending up with a fairly pristine C: partition with minimal dynamic data on it. In theory.

I’ve been deploying my SBS 2008 installs with a 100GB C: partition simply because I figured that over time, something would find a way to suck up all the space on C: and we’d eventually get to a point where we’d have to deal with resizing paritions or doing manual data cleanup. I didn’t expect that I’d hit that scenario just over a year after my first SBS 2008 production deployment.

In the last couple of weeks, my monitoring tools have started chirping about low disk space on C: on a couple of installs. Sure enough, one installation had 17GB remaining of a 100GB partition, another had 3.5GB remaining on an 80GB partition (my own production box, and yeah, it really needs an overhaul, but that’s another story). I started digging around and found the most common disk hog that’s been complained about across the net, the winsxs folder. Based on everything I’ve been able to read about winsxs, including a post from the Windows Server Core Team, that’s something that we’ll just have to live with, and really isn’t the point of this post anyway. Still, on my boxes, the winsxs folder still only amounted to about 12GB (bigger than what I’d like, but certainly not the primary culprit) which is only about 10% of my standard install C: space. Something else had been sucking away space and keeping it from me.

We use TreeSize from JAM Software as a standard utility on our server deployments to help monitor disk space usage, as this is something that comes up from time to time. [NOTE: this is not a specific endorsement of TreeSize, just a note that it's one of the many tools that we use in our operation.] So in the case of these low-free-space servers, I fired up TreeSize and went looking for the disk hog. Surprisingly, I couldn’t find it. I did clear up some areas that showed a larger-than-expected usage, but couldn’t find the smoking gun. A few weeks have gone by, and while I’ve been monitoring the state of these servers to ensure that free space didn’t get critically low, other tasks moved up on the priority list.

Then a discussion on one of my private lists cropped up regarding this exact topic, and I learned two valuable tidbits from that discussion.

The first is that in order for TreeSize to see the contents of ALL folders on the C: partition, it must be Run As Administrator. Upon reflection, this makes sense, but I know it’s catching a lot of experienced system admins off-guard. Some are advocating disabling UAC on the server to avoid this kind of issue, and I’m honestly not fully decided where I stand on that, so I won’t comment either way on that. But it does serve as a reminder that many system tools we may have been using for years on 2003 servers might not behave the same way under 2008 if you don’t use the almighty Run As Admin option.

The second is that the WSUS site in IIS has been logging an OBSCENE amount of data into the IIS logs folder. One of my servers had nearly 30GB (yes, that’s 30 gigabytes) of data in the WSUS log folder (C:\inetpub\logs\LogFiles\W3SVC1372222313). Another had just over 20GB. And in looking in the folder, I saw numerous DAILY log files that were well over 100MB each, with some well over 200MB each.

Once I cleared out the old log files (honestly, how far back am I going to need to look at WSUS logs anyway?) the free space on C: increased to a reasonable level, and my monitoring stopped yelling at me quite so often.

There are multiple lessons learned from this experience for me. The first is the whole reminder about Run As Administrator in the Server 2008 era. I’ve even taken to labeling some shortcuts with “Run As Administrator” in the icon name just to serve as a reminder. The second lesson is that 60GB is certainly NOT going to be sufficient as a minimum partition size on a production SBS 2008 server, even if all other data is moved off to different volumes (and I haven’t even covered the option of moving the WSUS SQL database files off of C: to another partition, which can’t be done through wizards but must be done by hand). With winsxs and the WSUS logs as two items that will definitely be grabbing disk space unexpectedly (well, it’s expected now anyway), we can be sure that over time there will be others. And as stated on the Core Team blog, you can only expect that winsxs will continue to grow over time. If it’s 12GB now, how large will it be in a couple of years? The third lesson is that some logging that happens automatically on the server probably should not just be left unchecked. If you enable SMTP logging (which I do and recommend for troubleshooting purposes), you should clean out old SMTP logs on a regular basis. Well, now you can add WSUS/IIS logs to that approach as well. There are numerous posts out there for ways to script this process, and I’m evaluating the approach we’re going to take within our operation to make this happen for our customer base.

If you’ve been struggling with low disk space issues on SBS 2008 C: partitions, hopefully this information will help you get a better handle on the immediate actions as well as the long term strategy that you’ll develop for your particular environment.