I Geek, Therefore I Am - Biztalk Deployment Framework and Biztalk 2006

If you will never in your life use Microsoft Biztalk Server (2004 or 2006), you should just quit reading now, after all, life is much too short. But, if you want to know what the hell I'm talking about before I get complicated, Biztalk Server (BTS) is a "service broker", it ties together web services, databases, file drop locations, flat files, SAP, Peoplesoft, et al, in a great big flow chart that can compensate for gaps in the process by queuing workflow up for a year or more if necessary. Now, on to what's really geeky.

Tonight I hit the Biztalk Users' Group at the Microsoft offices. This particular meeting was hosted by Scott Colestock, unarguably one of the shining lights of Biztalk development. His seminal contribution has been to make Biztalk deployment - the process of building your application and moving it to various environments - somewhat less than impossible. Don't get me wrong, you can do it without his Biztalk Deployment Framework (hereafter, BDF), and I, for one, have, but it's an amazing mishmash of assemblies, xml binding files, .bat files/scripts, Nant files, command line settings to differentiate environments and connection strings, virtual directory settings, permissions, orchestration/schema/mapping installations (which actually breaks down to GAC-ing vs. deploying if you're a Biztalk developer), etc, etc, etc, etc...just keep going, because whenever you think you're done tweaking the install, there's something you missed, not to mention you're usually doing the build/development under a copy of Virtual PC, requiring about 2 gigs of memory on your development machine.

Anyway, Scott has made the process significantly easier, automating the build into an MSI file and allowing deployment to finesse the settings so that they change on a per-environment basis. When I was doing a Biztalk install (and I'm probably the only guy in the history of the development who's learned Biztalk and walked away - it's just too expensive, developerwise, not to have your developers stay with Biztalk once they're good at it), I actually looked at Scott's setup, but it was early in the creation of the BDF and we'd already moved rapidly in a different direction. I spent 18 hours one Saturday/Sunday trying to shoehorn our hundreds of files backwards into Scott's setup style, only to decide that it wasn't worth my sanity. Do-able? Absolutely. But with code development constantly moving forward, I couldn't capture a large enough slice of time to freeze new features so that I could move everything into a different project structure (and, hence, namespaces).

So...what's so cool about the BDF that I would waste 18 hours of my life trying to undo a build/deployment process I'd already finished? For one, everything in Biztalk eventually relies upon something else. Not just assembly dependencies, but orchestration dependencies upon maps, and map dependencies upon schemas and orchestration dependencies upon other orchestrations and pipelines, which depend upon other schemas...when you've got piles of files in place, you have to make sure they're deployed in the appropriate order. Truly a headache. Scott's BDF breaks these dependent bits apart into separate projects so that they can always be built in the appropriate, dependent, order - basically MyBizTalk.Schemas, MyBizTalk.Mappings, MyBizTalk.Orchestrations, et al. Seems simple, but your average .Net developer tends to group things by project/application - i.e. these are the things I need to make this part of the project work, I shall group them as a cohesive unit - once you do that, life can get very ugly in the Biztalk universe, particularly if you're trying to deploy/redeploy without bringing down your app for a 24x7 business application (which is usually what BTS applications are trying to fulfill). There are often ways to cheat, which you can learn at Jon Flanders' blog, but they're often a.) not always obvious or b.) not always possible, or c.) not possible given the deployment process you've created.

So, back to Scott Colestock, why the BDF and how does it stack up against other options? There are basically four deployment options:
1.) BTSInstaller (in the BTS documentation) - creates an MSI, but no environment-relative settings/config changes and fairly painful for developers, particularly if they want to push it to their own machine.
2.) BTS2004 Batch File Deployment Zip - out at http://www.gotdotnet.com/team/wsservers/ - a bunch of scripts you can use to wrap what already exists. No MSIs, but a deployment zip and a more structured way of addressing all the scripts and binding files that you're probably already dropped a Nant construct around.
3.) Your own pile of scripts, binding files and various junk/doodads - basically #2, but you built it yourself. Inefficient in some respects, but at least you're assured that it absolutely fits your own needs.
4.) Scott Colestock's Biztalk Development Framework (BDF) - the end all of Biztalk deployment, if you haven't already started down the road of #1-#3. Why is it better than the rest? You get Visual Studio IDE integration, deployment tasks actually show up in the tools menu. If you bundle a package, it will also show up in the Windows menus under an entry for the application, allowing you to deploy, redeploy (remove and deploy), undeploy and test your Biztalk project (the later if you've created an NUnit testing assembly for your application, and might I suggest, if you're using .NET, that you create one of these. If you haven't, it's a hoot. And if you're in the Visual Studio IDE, you can actually tell the assembly to start up under an external program (i.e. NUnit) and debug without having to create the ubiquitous Console Harness application. While that alone is a bonus, the real power comes from being able to make your build/test group run the tests everytime the application is built. The more ideas you can come up with for ways to break your application, the more you can ensure that it doesn't break.

Ok, so, items in the tools menu...what else. We get the XML preprocessor. I was excited to hear about this bit o' code because it's a sourceforge open source project that allows you to insert conditional logic within your xml files. I'm currently mucking about with a bunch of configs for NUnit, and this will take care of testing my configs against all environments. The preprocessor allows you to insert code into your XML files that changes them based upon conditional logic by replacing various token values. Instead of having several binding files, you can have one that changes from environment to environment. But wait...not so fast. A heads up that if you have already created half a dozen binding files based on environment, you're positioned to really take advantage of Biztalk 2006 which will allow you to include several binding files in the resources folder and pick one upon MSI deployment. Your marked up, ready for preprocessing, Scott Colestock-style file will be just that much overkill, unless you're using his framework.

And it doesn't stop there. The BDF will deploy the Nant.exe assembly with itself so that your deployment folks don't have to install Nant themselves. Your .msi will haul along enough of Nant that it can deploy without a Nant install and .config/environmental variables tweaking.

The MSI deployment is open source as well, relying on Wix. If you get good at it, you can use it for other project deployments.

Whew....lot o' stuff.

And I learned some tricks that I hadn't thought of while I was working on my own deployment bundle. For instance, when deploying to a Biztalk farm, do the GAC boxes first, and the deployed box last. In a Biztalk farm, you only have to truly deploy once, that's when the orchestrations and schemas and maps and all are registered in the database. On every other box, each which already knows it belongs to the farm, you can get by with just putting your BTS assemblies in the global assembly cache. So when you're doing a farm install, do the GAC boxes first and the deploy box last, and you break the fewest boxes possible as you progress (if you deploy first, they all break). Also, and I reiterate, make NUnit tests. Biztalk is specifically designed to facilitate workflow, so create tests that simulate this workflow and make sure your workflow flows. Use those tests to call the endpoints of your SOAP/Filedrop locations and run an end to end after deployment so that you know everything is appropriately created and, in the hoary world of BTS permissions, appropriately configured.

Whew...what could be left. Biztalk 2006 of course. Released recently, but not truly available until next year. Don't be surprised, it has the year right in the damn title. What can we expect? According to Scott Colestock, not enough. The big change is that BTS2006 will support the idea of "applications". That is, instead of looking at your BTS project as a great big group, you can view it as distinct applications, bundled up as MSI files, with individual resources and a level of granularity that makes deployment more manageable. Wow...completely separate. Well...not quite. If you cross namespaces in a BTS group, even if in separate applications, that's no good. And applications can reference other applications, which is one more level of dependency to script. But you should be able to discretely deploy/undeploy applications that don't create dependencies. You'll be able to deploy whole solutions and BTS will be smart enough to manage the dependencies between ochestrations, schemas, maps, etc. This is a big change from BTS2004 and having to manually step through every single item if you weren't automated. Many items become resources in BTS2006. Extra .Net assemblies, custom functoids, scripts, policies - you can use the Admin MMC (unfortunately, not Visual Studio) to assign these items as resources. This is where you'd dump those environment-specific binding files you've created. When you deploy the MSI, it will actually prompt you for which binding file to use. However, BTS2006 will still strip out password-specific connection string settings, forcing you to manually change files if SQL or Oracle connections are part of your project/ports. BTS2006 deployment will distinguish between "import", actual deployment to the BTS database, and "install", simply gac-ing on a farmed machine. And finally, BTS2006 still won't provide a deployment success/failure log if deployed from BTS tools (you can do this by scripting and using the command line), which pretty much makes it worthless if it fails, because the amount of time it takes you to figure out where it went wrong is surreal.

So...I bet you'd like a summary. BTS2004: installation is evil. BTS2006: installation is still evil, but works if you have separated binding files and possibly no SQL ports. Scott Colestock's BDF: the bomb if you haven't already created several hundred files and a separate install. BTS itself : an excellent tool if you've had a year to familiarize yourself with the ins and outs and have access to half a dozen more seasoned developers who can tell you that what you're doing is not only not optimized, but idiotic.