October 22, 2009

Shame to me: DO4 installer does not properly sets up everything for IIS 7 and 64-bit Windows

That's true I discovered few days ago after post in our forum. The story behind is quite simple: I still use Windows Server 2003 (it's still alive ;) ) at my workplace and Windows Vista at home. Since I'm the person responsible for installer, I launch it and test mainly by my own, although I frequently ask the guys from my team to test it as well on their own PCs. So how this could happen that:
  • Nothing is properly installed on any 64 bit Windows. Basically, everything is copied, but nothing more!
  • The process of ASP.NET sample installation was actually changing nothing at all on IIS 7!
64 bit Windows support was broken mainly because of wrong detection of Visual Studio paths there and few an issue related to bracket handling in .bat files - I'll write about this later, but briefly, even echo "Something (x86)" may lead to an error there. Much worse case is if "%somePathWithBrackets%"=="" ... :)

ASP.NET sample installation was broken by much simpler reason: I used the same VBScript as for IIS 6 there instead of appcmd.exe. Shame for me :( Btw, I suspect I know why I didn't knew this: I have installed DO4 on home PC. Since installer is actually incapable of not just installing the sample, but uninstalling it as well, I was simply opening my own pre-configured version of it. "Voila - it works!"

The only thing surprising me is why no one wrote about all this stuff. We track all the downloads (likely, you know this - we send few notifications after each one). So I know how many persons have downloaded DO4 e.g. in last week, and I got just one complain...

Btw, we know documentation and installation are the worst problems people face - by our "exit polls". I admit I did may be 90% of what I could to improve the installation, integration with MSBuild and very basic usability during last month, so I hope these bugs are the last important ones there - today I'm planning to fix the last minor problem with ASP.NET sample that currently exposes itself on 64 bit Windows Server 2008 (probably, the issue is related to any IIS 7). The nightly build that is published right now installs everything properly, the only exception is this issue.

And finally, a single conclusion: I should always keep in mind which failures are the most expensive!

Earlier I wrote our testing process is very comprehensive: we're constantly keeping just few failing tests (mainly, for issues we're resolving right now) and pretty good code coverage. Take a look at TeamCity screenshots I made few minutes ago.

  

Imagine:

  • 36 build configurations running tests on all versions of all databases we support
  • ~ 1100 tests for Xtensive.Storage 
  • ~ 1000 tests for its dependencies
  • Pre-commit tests, 5 build agents and so on... 
  • But these test do not covers installer!

Yes, testing of installers is what really hard to automatize. But does good product testing matters at all, if its installer fails in may be 80% of cases because of just few bugs? I'm thinking, how costly may such a buggy installer be... "I have 64 bit Windows. Install was ok, but nothing works. So I uninstalled it and added the product and the company behind into in my personal blacklist." Likely, that's what really happened with 70% of our downloads in just first 5 minutes of their completion. Taking into account there is poor documentation as well, I can easily raise up the number to 95%. So that's the answer to my own question: "Why no one reports there are bugs in installer?".

I apologize for this huge mistake I made. I hope if you're reading this blog, you're the one of persons that did not blacklisted us because of this :( We'll publish completely polished installer in a day or so.