Windows Vista, spawn of Satan

I don't talk about work very much on this humble blog. In fact I almost never do. But for the past roughly two months I've been saddled with tweaking a Windows app so it works "properly" on Vista, and I need to vent a little. Ok, a lot. And not about obvious stuff like the ugly UI, or the way the OS badgers you to authorize and confirm every little thing you try to do. I turned most of that off right off the bat. No, once you elbow your way past the glossy nonsense and try to write some actual code, you run into a whole new set of annoyances.

• They've changed the event logs all around, for no obvious reason. In particular, they've tinkered with the security log and changed the event formats around, so any code that walks the log looking for logon/logoff events, filesystem activity, etc., is going to need to change. They haven't added much of anything useful that I've seen so far. At least, for "compatibility reasons", they've also changed the event IDs, bumping everything up by a nice round 4096. So that event #560 (request for privs on a file or other object) is now event #4656. Oh, and even though they've been working on Vista since around the time Win98 SE shipped, they don't have any formal documentation on the new formats just yet. Nice.
  
  Oh, and they've also added a whole new API for reading and writing to the event log, although you aren't actually obligated to use it. Yet.
  
  The only improvement I've noticed so far is that the security log will finally tell you which registry value changed, instead of just telling you the key name and making you guess. It's taken umpteen years and a day, but they finally fixed that. So that's hopeful in a way -- maybe, just maybe, all the other things that suck about Windows aren't permanent after all. But don't hold your breath waiting.
  
  
• MS quietly added some mandatory access control features, by kludging MAC data into objects' security descriptors as part of the SACL, where it doesn't really belong at all. And then they go and defeat the entire point of mandatory access control, by having all regular apps run under the High security label. Yes, even IE. Bastards.
  
  I realize it would've been a pain to add a new component to security descriptors, and I imagine they kludged it into the SACL because the OS protects that a bit more. But now Windows has to check two places instead of one (the DACL) to see whether you have access to a file or not. This probably isn't the main reason Vista is so slow, but I'm sure it can't be helping.
  
  Since this is a new feature in Vista, you can't reasonably expect it to be supported in Explorer. And if you did, you'd be wrong, because it's not there. Even though it's physically part of the SACL, you only get it back if you also specify the new LABEL_SECURITY_INFORMATION flag when querying an object's security descriptor. Naturally Explorer doesn't do that. At least there's a command line tool (icacls.exe) you can use if you need to.
  
  In fairness, M$ hasn't been promoting this at all, so they may realize it isn't ready for prime time just yet. If they handle it like they've handled other filesystem features, they may let it sit for a decade or so before making it do something useful (like hardlinks), or they may just abandon it in place, leaving mysterious stubs here and there around the OS (like COM-style "structured storage", or OS/2-style "extended attributes"). Or it'll just remain in limbo for eternity, present but not easily seen or used, like alternate data streams.
  
  
• Oh, and there's always folder virtualization, where your app thinks it's writing a file to Program Files, but Vista lies to you and silently redirects the file to somewhere else, so as to keep Program Files all pristine and whatever. It turns out that you can sort of turn this off. But unlike the 32 vs. 64-bit virtualization that appeared in 64-bit XP, there's no brain-melting API to toggle this on and off for your app. No, instead you mark the app as requiring administrative rights, and you do this with an XML file called a "manifest". That gets compiled into a string resource that Vista looks at first when deciding whether to run your app or not.
  
  So the plus side is that your files really do go where you put them. The minus side is that you've got to do the "Run as Administrator" dance first, with scary UAC dialogs galore. Unless you turn off UAC. Signing your app with a nice, expensive Authenticode cert will give users a somewhat less scary UAC dialog, for whatever that's worth.
  
  
• Someone ought to sue MS for multiple abuse of the word "Virtual". On top of at least two unrelated kinds of folder virtualization, and at least one kind of registry virtualization, they've also added a new file attribute bit, "FILE_ATTRIBUTE_VIRTUAL", which MSDN merely describes as "The file is a virtual file." Which is not overly helpful. I'm pretty sure it's unrelated to the Program Files thing in the previous item, but I still have no idea what this is for, or whether I need to worry about it. Bastards.
  
  
• Did you know Vista supports hard links and symlinks now? It's true! Under the hood, symlinks are implemented with complex entities called "reparse points", and there are six other kinds of reparse point, including directory junctions, mount points, DFS mount points, Hierarchical Storage Management doohickeys, Single Information Store ( a sort of fancy distributed copy-on-write symlink-ish arrangement), and much, much more. Hard links, on the other hand, are a low-level NTFS feature implemented with alternate data streams, more or less. Don't get me started about alternate data streams. Actually hard links have been around since the NT 3.1 days, but MS didn't bother to document them much until now.
  
  Kind of a shame that Explorer doesn't know squat about directory junctions, even the ones created as part of the Vista default install (like "C:\Documents and Settings", which now points to "C:\Users", for example).
  
  
• If you're feeling brave, install SUA ("Services for Unix-based Applications") on Vista. Enable case-sensitive filenames during the install, open a shell window, and create a couple of files whose names differ only by case. Then try opening either one in Explorer and see what happens. Snort. Giggle. This isn't exactly new behavior or anything, but the way they've been pushing this SUA stuff lately you'd think they'd have taken the time to iron out certain longstanding kinks.
  
  
• I actually ended up deactivating SUA, because it didn't get along with Visual Studio 2005. I never quite figured out the details, but when SUA was enabled builds would either stall out, take forever to run, or fail with unspecified errors. But be warned, if you deactivate SUA, you also need to disable the Posix debugging plugin within VS2k5. Otherwise the IDE will explode whenever you try to debug an app, any app, not just Posix ones. It's kind of a shame really; I was starting to warm up to SUA. I'd installed gcc and almost got bash working when I realized it simply wasn't to be. Oh, well.
  
  
• Yet another obscure and poorly documented tidbit: Vista added a new registry function called RegLoadAppKey, which basically lets apps create their own private chunks of registry that aren't visible in Regedit, because they lie outside the usual HKEY_LOCAL_MACHINE / HKEY_USERS / etc. keys. The Windows kernel uses a different namespace than mere mortal apps do, so to it HKEY_LOCAL_MACHINE is "\Registry\Machine", HKEY_USERS is "\Registry\Users". C:\ is usually something like "\Device\Harddisk0\Partition1", but I digress. When you create a private registry hive, it mounts under "\Registry\A\{GUID}", where {GUID} is a unique ID generated new each time a hive is loaded. If spyware isn't already exploiting this, it's only a matter of time. Granted, you could already mount a hive under \Registry if you knew your way around the native API, but now they've made it so easy even a script kiddie can do it.
  
  I don't know for a fact that private hives always show up under "\Registry\A", since the whole thing's totally undocumented. But that's the behavior I've seen so far.
  
  
• Transactional NTFS isn't an inherently terrible idea. I can see cases where you'd want to be sure either all of your file & registry changes happen or none of them do, so that your app data doesn't get wedged in an inconsistent state. It's kind of a shame they had to introduce new "transacted" forms of all the existing file & registry functions just to support this. (Actually I'm not sure they did all of them, because I haven't checked.)
  
  It'll probably be years before we get a handle on all the oddities and implications around doing access on a transactional model, but I've seen at least one thing already. If you, say, delete a file as part of a transaction, the audit event for the delete includes a "transaction ID", which is just a big ugly GUID (this being Windows and all). Then when you commit or roll back a transaction, another event shows up in the log saying the transaction state has changed, giving you the transaction ID and the new state, which is an opaque integer. If I recall correctly, "commit" is 52 and "rollback" is 56. Or perhaps it's the other way around, I forget now. So if there's a rollback, you have to go back and look at all the activity you saw earlier tagged with that transaction ID and treat it as if it never happened, basically. It's always an adventure... Yeah...
  
  

Before starting this post, I spent a fair amount of time carping about Vista elsewhere. I figured I'd copy-n-paste those comments here so I can concentrate all my whining in one place....

---

From IV

So at the office I just (sorta) wrapped up a Linux coding project that's been sucking up most of my available cycles for a while now. And fresh of that, I've been tasked with tweaking a different app to make it play nice with Vista. Understand that 90% of my coworkers are Java & web folks who know zilch about underlying operating systems, and I wear both the Unix and Win32 guru hats in these parts. Which is good, most of the time. It certainly makes for great job security.

It's even convenient to start on Vista now because I just got a shiny new Core 2 Duo box, which may be just barely fast enough to run Vista halfway decently. But there's a problem. Corporate IS has decreed that Vista is Verboten companywide. I can't blame them, of course, and I think I actually crowed about that policy here once upon a time, but now it's becoming a problem. The official line is that I *can* install it on my new box, but if I do I can't connect it to the network. I may have to go to IS in the near future, hat in hand, and beg them to bend the rules just this once on my behalf. For stupid freakin' Vista, of all things.

[ I've used Vista exactly once before, while trying to fix my mother's fancy new HP touchscreen machine. Ten minutes of that, and I concluded that Vista is evil and everything about it is wrong. But mom likes it because of the flashy graphics. When I told her my home is Windows-free, she looked at me like I was from Neptune. But at least I fixed the problem, which turned out to be the lack of a USB cable between the computer and the printer. :) ]

Again from IV:

So, as you might be aware, I'm stuck with Vista, and every day it sucks worse than the day before. Today I came in to the office and noticed Firefox had gone haywire, in a way I haven't seen it do on any other OS. Somehow it had picked up a memory leak, and was using 1.4GB memory and 50% cpu. Or at least that's what Vista said it was doing. Possibly this is a genuine problem in FF, but it's also possible a certain predatory OS+browser monopolist is messing about with a certain competing browser. It's happened before, and nothing would surprise me at this point.

But that's not the really cheesy part. I went ahead and killed FF, and Vista -- the OS with the fancy gee-whiz graphics and all -- can't seem to figure out that I no longer require a grey outline of where my FF window used to be. It's not just an artifact of lazy screen drawing or anything; it redraws over top of all other windows. You move something around, and the big grey outline is still there. I thought they'd sorted out this crap back around Windows 2.11 or so, but apparently not.

"Fortunately", Windows Update has downloaded the latest set of critical bug fixes and security patches and such, and every few minutes it pesters me to reboot. So reboot I shall.

Stupid Vista.

More of the same. A little redundant, and not all of it Vista-specific, but hey.

Ah, the weird, wonderful world of Windows filesystem oddities. As I've mentioned before, I'm currently in the midst of tweaking an existing Windows app so it runs properly on Vista, so I've been forced to remember a bunch of ugly details I'd happily paged out.

A few of my favorites, just off the top of my head:

• If you want to know the access & write times on a file, the value you see depends on how you ask. My theory is that Windows flushes metadata updates to disk on a very lazy, periodic basis, although I don't believe they've ever documented this. GetFileTimes() appears to return the in-memory values, in the disk cache or somewhere, while FindFirstFile/FindNextFile always hits the disk and returns whatever's there. I've seen the two out of sync by up to an hour at times. Naturally there's an obscure registry setting that lets you turn off updating of file access times entirely, and this seems to be the default on Vista. Although again, this is completely undocumented.
  
  
• Filenames are not case sensitive, except when they are. There are several ways to get two files in a directory whose names differ only by case: Unix drives shared via Samba, or files created under Services for Unix, or files created with plain old CreateFile() with the FILE_FLAG_POSIX_SEMANTICS option (although the latter doesn't work by default on XP & later). Explorer gets deeply confused when you try to open one of these, and opens whichever comes first alphabetically (I think), regardless of which one you actually clicked on.
  
  
• Paths are limited to MAX_PATH (260) characters, except when they aren't. 260 characters too restrictive? Just prepend "\\?\" to your path and away you go, up to around 32000 chars. Except that nearly all Windows apps don't know about this trick either, so you can create directory trees that almost nobody can see into. To be fair, you can do something similar on Unix too. It's easy to write a little program that creates a directory with a long name, chdir's into it, creates a subdirectory with a long name, chdir's into it, and so forth, and do a getcwd() at each iteration. Eventually getcwd() starts failing, but the OS is quite happy to go on creating subdirs for you until you fill up the disk.
  
  
• The registry has a different trick for creating keys nobody can see into. The underlying (and little-known) Native API refers to object names with counted strings, not null-terminated ones, so you're free to use embedded null characters in the names of things. Well, some things. I don't think NTFS accepts null chars in names, at least. But the registry's just fine with it. The regular Win32 API everyone uses *does* use null-terminated names, though, so if you create a key with an embedded null in the name, there is simply no way to open it with RegOpenKeyEx(), because its name is basically "unspeakable" as far as Win32 is concerned. MS actually used this trick to secure sensitive bits of the registry, like parts of the local SAM db. Which works great so long as nobody learns the trick. Security through obscurity...
  
  
• Vista introduced a new twist on "unspeakable" registry names. There's a new function called RegLoadAppKey(), which promises to give your app its own private chunk of registry nobody else can see. And it's true, you get a bit of registry that can't be located with HKEY_LOCAL_MACHINE, HKEY_USERS, or any of the other predefined keys.
  
  A bit of background is in order here. The NT kernel uses a different path scheme than what people are used to seeing in Regedit, or Explorer for that matter, and the names of things you encounter are not their *true* names. HKEY_LOCAL_MACHINE is really \Registry\Machine, HKEY_USERS is \Registry\Users, C:\ is usually something like \Device\Harddisk0\Partition1, and so forth. I rather like this unified namespace, actually, as it makes things feel just a little more Unix-like. So of course it's completely invisible to end users, with occasional exceptions such as registry audit events.
  
  So the new trick from Redmond is to mount "private" registry hives under \Registry\A\{GUID}, where GUID is newly assigned each time a hive is mounted. Once you know where to look, "private" isn't quite so private anymore. Still, Regedit and other apps using Win32 functions can't see this stuff, so I expect it's only a matter of time before the spyware crowd starts exploiting this "feature".
  
  
• Vista now offers symlinks, which are not the same thing as hardlinks, or directory junctions, or shortcuts. Symlinks are implemented as "reparse points", and the symlink data is stored as an alternate data stream. Directory junctions are "reparse points" too. Actually there are at least seven different kinds of reparse point, and counting, and there's a complex API that lets you define your own types too, like you'd really want to do that. Hardlinks are a low-level NTFS feature that's been around since the beginning, although poorly supported until recently. Shortcuts are a higher-level feature, belonging to the UI part of the OS, and are implemented with COM.