BizTalk Deployment with MSBuild without Custom Task

In this post, I will be discussing more on BizTalk Deployment to production environment.

I just had a deployment yesterday, the scenario :
We have some bug fixes to some orchestrations which we need to apply to an existing BizTalk application in production.

I have been interested in automated BizTalk deployment since I read a post about Creating highly-repeatable builds and deployments from Brian Loesgen. However, Brian's post is about compilation and export of the application to MSI, and he's using a plain batch command file.
Later I found a post about Using MSBuild to Build and Deploy BizTalk 2006 Solutions from Stephen W. Thomas. The post is more relevant to my scenario, however I found that there are some custom tasks being used to perform the deployment.

I'm trying to make things simpler by using the existing tools because we have a quite strict policy for deployment and I hope with this, others can also use it without having to dig in into the custom tasks.

So based on both articles, I managed to come out with a set of MSBuild files for the deployment.

1. Deploy.bat : Batch command file for the IT person to run the MSBuild with it parameters (project file & log file)
2. Build.proj : Main project file which specifies all the steps (targets) to execute
3. Build.properties : Definition of variables & collections for the deployment
4. Deploy.targets : List of steps (targets) for the deployment

Grab the files from here :


Scenario :

1. Stop BizTalk Host Instance
2. Register Helper library to GAC
3. Stop & Unenlist Orchestration
4. Import MSI
5. Install MSI
6. Import Binding File
7. Start & Enlist Orchestration
8. Start BizTalk Host Instance
9. Copy MyWebServices update files
   

Deploy.bat

@ECHO Starting up MS Build
C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe Build.proj /l:FileLogger,Microsoft.Build.Engine;logfile=Build.log;append=false;verbosity=diagnostic;encoding=utf-8

Build.proj

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="StopBizTalkServices;RegisterGAC;StopOrchestration;DeployMSI;InstallMSI;ImportBindingFile;StartOrchestration;StartBizTalkServices;DeployWS">
<Import Project="Build.properties" />
<Import Project="Deploy.targets" />
</Project>

Note: Specify the steps (targets) to execute at DefaultTargets attribute value

Build.properties

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<!-- Path to the .Net utilities (gacutil.exe) folder -->
<DotNetPath>C:\Program Files\Microsoft.NET\SDK\v2.0\Bin\</DotNetPath>
<!-- Path to the BizTalk folder -->
<BizTalkPath>C:\Program Files\Microsoft BizTalk Server 2006\</BizTalkPath>
<!-- Path to the deployment root folder -->
<DeploymentPath>C:\DeploymentFolder\</DeploymentPath>
<!-- Path to the Web Services Deployment path -->
<WSDestinationPath>C:\Inetpub\wwwwroot\MyWebServices\</WSDestinationPath>
<!-- Path to the build dll files -->
<GACPath>$(DeploymentPath)\Common Library\</GACPath>
<!-- Path to the msi files -->
<MSIPath>$(DeploymentPath)\MSI Files\</MSIPath>
<!-- Path to the binding files -->
<BindingPath>$(DeploymentPath)\Binding Files\</BindingPath>
<!-- Path to the WMI script files -->
<WMIScriptPath>$(DeploymentPath)\WMIScripts\</WMIScriptPath>
<!-- Name of the BizTalk Application to deploy code into -->
<BTApplicationName>MyBizApp</BTApplicationName>
<!-- Bts Log File -->
<BtsLogFile>BtsTask.log</BtsLogFile>
<!-- MsiExec Log File -->
<MsiLogFile>MsiExec.log</MsiLogFile>
</PropertyGroup>

<!-- List all .net items that need to be GACed -->
<ItemGroup>
<GacStuff Include="MyLibrary1.dll" />
<GacStuff Include="MyLibrary2.dll" />
</ItemGroup>
<!-- List all MSI Files -->
<ItemGroup>
<MSIFile Include="MyBizApp.msi" />
</ItemGroup>
<!-- List all affected BizTalk Services to restart -->
<ItemGroup>
<BizTalkServices Include="BizTalk Service BizTalk LOCAL : BizTalkServerApplication" />
</ItemGroup>
<!-- List all web services files to deploy -->
<ItemGroup>
<WSFiles Include="$(DeploymentPath)\MyWebServices\**\*.*" />
</ItemGroup>
</Project>

Note: I grouped the deployment files into certain folders to make it easier to understand :

1. Common Library folder -> Consists of helper dlls which needs to be GAC-ed seperately
2. MSI Files folder -> BizTalk generated MSI Files to import and to install
3. Binding Files folder -> Binding Files to import
4. WMIScripts folder -> Consists of vbs files to stop (StopOrch.vbs) & start (EnlistOrch.vbs) the orchestrations, you can find the vbs files under the C:\Program Files\Microsoft BizTalk Server 2006\SDK\Samples\Admin\WMI
5. MyWebServices folder -> Consists of updates for the existing MyWebServices

Deploy.targets

<Project  xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<!-- START DEPLOYMENT -->

<!-- Stop BizTalk Services -->
<Target Name="StopBizTalkServices" >
<Exec Command="NET STOP "%(BizTalkServices.Identity)"" ContinueOnError="true" />
<Message text="Stopped BizTalk Services : %(BizTalkServices.Identity)" />
</Target>

<!-- Register GAC -->
<Target Name="RegisterGAC" >
<Exec Command=""$(DotNetPath)gacutil.exe" /i "$(GACPath)%(GacStuff.Identity)" /f" />
<Message text="Registered DLLs to GAC : %(GACStuff.Identity)" />
</Target>

<!-- Stop Orchestration : The process sequence matters because of the dependency -->
<Target Name="StopOrchestration" >
<Exec Command="cscript.exe $(WMIScriptPath)StopOrch.vbs [Orchestration name 1] [Assembly Name 1] Unenlist" />
<Exec Command="cscript.exe $(WMIScriptPath)StopOrch.vbs [Orchestration name 2] [Assembly Name 2] Unenlist" />

<Message text="Stopped and Unenlisted Orchestrations" />
</Target>

<!-- Deploy MSI -->
<Target Name="DeployMSI" >
<Exec Command=""$(BizTalkPath)BtsTask.exe" ImportApp /Package:"$(MSIPath)\%(MSIFile.Identity)" /ApplicationName:$(BTApplicationName) /Overwrite >> $(BtsLogFile)" />
<Message text="Deployed $(BTApplicationName)" />
</Target>

<!-- Install MSI in quite mode -->
<Target Name="InstallMSI" DependsOnTargets="DeployMSI">
<Exec Command="MsiExec.exe /i "$(MSIPath)\%(MSIFile.Identity)" /quiet /l+ $(MsiLogFile)"  />
<Message text="Installed %(MSIFile.Identity)" />
</Target>

<!-- Import Binding File -->
<Target Name="ImportBindingFile" DependsOnTargets="DeployMSI">
<Exec Command=""$(BizTalkPath)BtsTask.exe" ImportBindings -Source:"$(BindingPath)$(BTApplicationName).BindingInfo.xml" -ApplicationName:$(BTApplicationName) >> $(BtsLogFile)" />
<Message text="Imported Binding for $(BTApplicationName)" />
</Target>

<!-- Start Orchestration : The process sequence matters because of the dependency -->
<Target Name="StartOrchestration" DependsOnTargets="DeployMSI;ImportBindingFile">
<Exec Command="cscript.exe $(WMIScriptPath)EnlistOrch.vbs [Orchestration Name 2] [Assembly Name 2] Start" />
<Exec Command="cscript.exe $(WMIScriptPath)EnlistOrch.vbs [Orchestration Name 1] [Assembly Name 1] Start" />

<Message text="Enlisted and Started Orchestrations" />
</Target>

<!-- Start BizTalk Services -->
<Target Name="StartBizTalkServices" DependsOnTargets="DeployMSI;ImportBindingFile;StopOrchestration">
<Exec Command="NET START "%(BizTalkServices.Identity)"" ContinueOnError="true" />
<Message text="Started BizTalk Services : %(BizTalkServices.Identity)" />
</Target>

<!-- Update the Web Services -->
<Target Name="DeployWS">
<Copy SourceFiles="@(WSFiles)" DestinationFiles="@(WSFiles->'$(WSDestinationPath)%(RecursiveDir)%(Filename)%(Extension)')" />
<Message text="Deployed Web Services" />
</Target>

<!-- END DEPLOYMENT -->
</Project>

Note: As you see above, I only use existing tasks such as exec and copy for the deployment and I use BtsTask.exe for BizTalk deployment. For the complete list of MS Build tasks, you can find it here.


After executing the deploy.bat, there will be 3 log files :

1. Build.log -> MSBuild generated log file which contains all the deployment activities
2. BtsTask.log -> BtsTask.exe generated log file for BizTalk deployment
3. MsiExec.log -> MsiExec.exe generated log file for BizTalk Msi File installation

Notes:

• GACUtil.exe may not be available in the server if the SDK is not installed, I did it manually using the Microsoft .NET Framework 2.0 Configuration yesterday.
• cscript.exe will not have return success or fail to MSBuild so it will not stop when there is any error.
• There is one tweak on the EnlistOrch.vbs, by default it will try to start and enlist the orchestration with the default host. Since I'm not using the default host for my application, i comment out the line 90 -> Inst.Enlist(HostName) and use only Inst.Enlist
• I haven't tried yet, but the Build.properties and Deploy.target files should be able to be merged into the Build.proj itself, however by splitting them into 2 files, they are easier to maintain and understand as well.

Why bother ourselves to use MSBuild than to use plain batch command for this like Brian's post?

1. MSBuild provide built-in file log mechanism for the whole execution process
2. Better and clearer dependency specification between tasks
3. MSBuild comes with .Net Framework 2.0 which means that every BizTalk server should have MSBuild installed without any additional installation efforts.
   

My wish list :

1. MSBuild provides new Built-in task for GACUtil
2. BizTalk Server provides add-on BizTalk tasks for MSBuild
   

Please leave questions or comments if you have any ;)
Hope this helps :)

User Interaction, Behavior, and Feedback

Warning : Non technical stuff content ;)

I stumbled upon Jeff Atwood's blog post on UsWare vs. ThemWare 2 days ago. The terms was originally from Eric Sink's blog post on Yours, Mine and Ours. It is a good advice and a very relevant on what we do today as a software developer.

User Interaction
In the first project as a software developer, I was indeed developing software as ThemWare. Why? Because I only knew how to code and test the software with my very own perspective which was more less as long as this page is working (able to display, save, edit, and delete), I'm done with it :p
Then as the program went live, I was there sitting beside the user using the application to make sure everything is going fine.
Well, guess what, the application caused a very long queue of the customers in peak hours and i could see from their faces that they were not tolerant for waiting that long :p
Reason : it was a new application for the users, even though they had the trainings but the load are obviously different compared to peak hours, "panic" mode is ON and made things worse when you can't do things as fast as before using the new application.
At the end of the day, I got some "major" feedbacks from the users which i can clearly understand because i was sitting with them whole day :
1. The page navigation was bad, too many steps to follow in order to do a transaction
2. All have to be done by mouse, they need keyboard short cuts and use tabs so they can do one transaction faster.
3. Too many post backs in the page and the page is too long, so they have to scroll down again and again.

So I did a quick major changes based on those things and that really helped the users afterwards and i was really glad that my program can really help to increase their productivity, that is why we create softwares right? To increase job productivity and efficiency :)

From those experiences, i keep trying to also see things from a user perspective when developing applications, making it as easy as possible to do things.

So don't just make the things work, but make them USABLE as well by creating the UsWare software ;)

User Behavior
I believe that the assumption that the user will use the program as we expect is totally WRONG. For example, as the developer of the program X, I certainly know that after I do (A), I won't do (B). But that may not apply to the users, if we don't put any validations or clear steps to prevent that, it is always a big chance that they will actually do that ;p
It is always hard to predict the user behavior, but adding some thoughts of the variation of possible use case scenarios will be very helpful :) It will be much more costly to handle problems when it happens in production later.

User Feedback
I feel that it is very important to listen to user feedback, because they are the one who are using the software. However, do make a deep consideration or have some discussions first before following and making any changes.
Reasons :
1. There are some cases where after a change has been done, then turns out that it makes things worse or at least not helping.
2. YAGNI (You ain't gonna need it) or you ain't gonna use it
But in any cases, if you receive feedbacks from the users, that means a good thing, they are really using the application :)

For me, I always feel really happy when a user come back to me and said "Thank you, you just make my life easier with that" :)

Regular Expression / RegEx with Expresso

I'm not a RegEx expert :P I'm using RegEx for some simple validations, creating scripts and editing some string.

I remembered the first time i read a RegEx expression a couple of years ago, i said to my self, "WOW, this really looks like an encrypted string rather than an expression" :P

What i want to share here is that several months ago, I stumbled upon Expresso, it is a development tool for regex, this is a powerful stuff, there are tons of features and library resources inside and the authors from Ultrapico is kind enough to give away the tool for free :)

So nowadays, every time i need to create a RegEx expression, i always open up Expresso, test that my RegEx expression (plus the replacement string) is working as expected, then i deploy the expression into the code. Testing the results is much easier and faster than ever ;)

If you do need to work with RegEx, i really suggest you check it out ;)