BizTalk Server 2006 Documenter

While browsing my feeds through google reader, I found a post from Brian Loesgen, he posted a lot of very useful links related to BizTalk. If you're a BizTalk developer, i really really encourage u to take a look at it ;)

One of the link which interest me is the BizTalk Server 2006 Documenter, because I'm currently doing some documentations for the recent BizTalk applications which we just have completed.
From the release date, it seems that this tool has been a while actually, but hopefully it will be helpful for you guyz who are just like me ;)

This is a direct link to the codeplex site.

The result is really really awesome, you can find the whole information that you can see while you're designing the orchestration, such as : Messages, variables, even the orchestration design (Gosh, just thinking how much efforts i did previously to put them in the my own documentation) and the information in the BizTalk server, such as : applications, orchestrations, ports, assemblies, schemas, etc.

Orchestration Information

Orchestration Design (This really rocks)

Orchestration Codes

Note :

• The Ports information will not have the detail properties for the adapters (SOAP Address, OracleDB, Properties)
• If you ever find this exception "There was an error generating the XML document" when generating the documentation, you might want to enable the error logging in the config file to dump the error stack trace to the log file. See whether this solution can solve the problem like it solved mine.

Documentation has never been as easy as this before ;)

Once again thanks a lot to the BizTalk Server 2006 Documenter Team.

Hope this helps :D

BizTalk Oracle Adapter / ADO.Net Limitation (ORA-01460: unimplemented or unreasonable conversion requested)

System defined exception
Call to RP_BIZTALKTEST@OracleDb://ORADEV/DBO/Procedures/TopLevel
Implementation = OracleDb://ORADEV/DBO/Procedures/TopLevel

Source: OracleDb
Error Code: 1460 (0x5b4)
Cause: OracleDb://exception=DBException (Unique ID )
42000 : [Oracle][ODBC][Ora]ORA-01460: unimplemented or unreasonable conversion requested


Exception data:
struct DBException =
WideString StatusCode = "42000"
WideString Description = "[Oracle][ODBC][Ora]ORA-01460: unimplemented or unreasonable conversion requested"

I'm using BizTalk 2006 with BizTalk Oracle Adapter to Oracle9i Database.

What i was trying to do here is to call an oracle procedure to insert a new row in a table and one of the field is CLOB/XMLType. This field is actually storing the current message which was passed in to the orchestration and to be used in future processing (other orchestration).

For simple / small size message, it was working fine as expected. Fortunately, I did try the scenario for huge incoming message (always test your code with the highest value of the assumptions (e.g. The user will be able to submit 1000 items in 1 transaction). Then I got this exception from BizTalk.

I tried to search through the net about this, it seems there is a limitation to pass in a parameter more than 32 KB size. (http://www.google.com/search?hl=en&q=32+kb+xmltype+parameter+procedure+limit&btnG=Search&meta=)

Some Tests which I did :
1. Use NativeSQL insert into table. - Not Working
XMLType column will be generated as base64binary type in the schema.
I have tried to pass the xml raw string but failed. Same if I tried to convert the xml into base64binary.
The error message which was returned :

System defined exception
Call to Insert@OracleDb://ORADEV/DBO/Tables/BIZTALKTEST
Implementation = OracleDb://ORADEV/DBO/Tables/BIZTALKTEST

Source: OracleDb
Error Code: 932 (0x3a4)
Cause: OracleDb://exception=DBException (Unique ID )
HY000 : [Oracle][ODBC][Ora]ORA-00932: inconsistent datatypes: expected NUMBER got BINARY


Exception data:
struct DBException =
WideString StatusCode = "HY000"
WideString Description = "[Oracle][ODBC][Ora]ORA-00932: inconsistent datatypes: expected NUMBER got BINARY
"

Or (I tried some combinations for the data type : CLOB/XMLTYPE)

Error transmitting message: SOAP-ENV:ServerRequest ID: Unknown
Exception Type: A cross object system conversion error occurred
Exception Info:
Exception occurred:
E-CORE0053: Dimensions or bounds of source and target params do not match
Call to Insert@OracleDb://ORADEV/DBO/Tables/BIZTALKTEST
Implementation = OracleDb://ORADEV/DBO/Tables/BIZTALKTEST

SOAP://xmlns/schemas.microsoft.com/BIZTALKTEST/InsertRecord:Type/[]/TESTRAW ==> OracleDb://ORADEV/DBO/Tables/BIZTALKTEST/InsertRecord/VARCHAR2/WideString
(SOAP://xmlns/schemas.microsoft.com/BIZTALKTEST/Insert:Element/Rows ==> sequence<struct OracleDb://ORADEV/DBO/Tables/BIZTALKTEST/InsertRecord&>)
(Error occurred before call to foreign method)


2. Call oracle Procedure passing XMLTYPE parameter - Not Working
I tried to use XMLType as a parameter but there was an error similar to the 2nd error above.

Error transmitting message: SOAP-ENV:ServerRequest ID: Unknown
Exception Type: A cross object system conversion error occurred
Exception Info:
Exception occurred:
E-CORE0053: Dimensions or bounds of source and target params do not match
Call to RP_BIZTALKTEST@OracleDb://ORADEV/DBO/Procedures/TopLevel
Implementation = OracleDb://ORADEV/DBO/Procedures/TopLevel

SOAP://xmlns/schemas.microsoft.com/TopLevel/RP_BIZTALKTEST:Element/V_TESTRAW ==> OracleDb://ORADEV/UNDEFINED_2000/WideString
(SOAP://xmlns/schemas.microsoft.com/TopLevel/RP_BIZTALKTEST:Element/V_TESTRAW ==> OracleDb://ORADEV/UNDEFINED_2000)
(Error occurred before call to foreign method)

3. Call oracle Procedure passing CLOB parameter and then convert into XMLTYPE when insert - Not Working
This is the one that actually has 32KB limitation. I actually created a small .net application using oracle client to test this and turned out that I got the same error when i tried to pass > 32 KB string into the parameter. There is a work around in this article, but i find it more complex than the 4th solution below (http://msdn2.microsoft.com/en-us/library/cydxhzhz(VS.80).aspx)

System defined exception
Call to RP_BIZTALKTEST@OracleDb://ORADEV/DBO/Procedures/TopLevel
Implementation = OracleDb://ORADEV/DBO/Procedures/TopLevel

Source: OracleDb
Error Code: 1460 (0x5b4)
Cause: OracleDb://exception=DBException (Unique ID )
42000 : [Oracle][ODBC][Ora]ORA-01460: unimplemented or unreasonable conversion requested


Exception data:
struct DBException =
WideString StatusCode = "42000"
WideString Description = "[Oracle][ODBC][Ora]ORA-01460: unimplemented or unreasonable conversion requested"

4. Use Oracle Data Provider for .NET 9.2.0.7.00 (ODP.Net) - Working
I found that this is actually the solution and the easiest way to do, passing a XMLTYPE is just as simple as passing other parameters, It has its OracleDb Type.XmlType. So far I have tested with huge messages more than 32 KB and they are working fine. Please also note on the ODP.Net version.

OracleParameter pContent = new OracleParameter("content", OracleDbType.XmlType, content.Length);

Before I decided to go with the ODP.Net, I have tried using :
1. ODBC Connection (System.Data.Odbc) : Limited with 4 KB size
2. .Net Oracle Data Provider (System.Data.OracleClient) : Limited with 32 KB size as well.

BizTalk Consume WebServices with SOAP Header

When working with Web Services, the common scenario is to pass in the credential in the SOAP Header.

In this post, I will provide the steps how to do this in BizTalk Orchestration.
This will be a simple flow like below screen shot :

Flow Detail :

1. Web Application will call an exposed web services by Biztalk Orchestration
2. The request object consists of 3 elements : UserName, Password, Message
   
3. BizTalk Orchestration will construct the SOAP Header Message with UserName and Password
4. BizTalk Orchestration will assign the Message as the body and assign the SOAP Header
5. BizTalk Orchestration calls the Web Services (WSSOAPHeader)
6. Web Services will read the SOAP Header message and return back some message
7. BizTalk will also return back the message to the web application

BizTalk Orchestration layout

One of the most crucial thing in here is the SOAP Header message.
To create the SOAP Header message schema :

1. Create a property schema file
2. In the schema properties, change the Target Namespace to http://schemas.microsoft.com/BizTalk/2003/SOAPHeader Set the Node Name property to the SOAP Header Name and the Property Schema Base to MessageContextPropertyBase

After creating this property, we can assign the SOAP Header into the message like this below

varXMLSOAPHeader = msgSOAPHeader;
msgWSRequest(MyBizTalkApp.AuthenticationSOAPHeader) = varXMLSOAPHeader.OuterXml;

Note :

• When assigning the SOAP Header, it expects string value, that's why I assign the msgSOAPHeader into an XML document variable and get the string from OuterXml property.
• If the SOAP Header is not received by the web services even though you have assigned it in the code, make sure that you have followed each of the steps to create the SOAP Header message schema above, especially the namespace part. It caused me 2 days to notice this one in the end :P

You can find and download the whole VS 2005 solution at here :


Here is the link to the MSDN Site : Consuming Web Services with SOAP Headers

Hope this helps

SSODB Configuration Store for BizTalk Orchestration

One of the aspect of the application design is the configuration module.

There are at least 4 repositories where we can store the configuration for BizTalk orchestration :

1. Custom config file
   
2. BizTalk config file
3. Windows Registry
   
4. Database

Since each company will have its own policy for the production environment, this will eventually lead into the configuration repository.

Each repository above has its own pros and cons to implement, but I would say that configuration files and registry will not scale for MultiServer / Clustering environment, since we need to replicate them to each of the BizTalk server. So it will be easier to use database since we can put the configuration information in one place and all the servers will access this central database.

But when it comes to the database, we will have the next question of where to store the connection string. Well, this is the point where SSODB comes in handy.

As we may notice, whenever we install BizTalk Server, the Enterprise Single Sign-on will need to be installed as well, and the Enterprise Single Sign-on has a database, named SSODB. This is mainly used for BizTalk to store all the internal configuration information.

This SSODB will be the ideal solution for the configuration repository since as far as i know.
Benefits :

1. SSO will be installed and running along with BizTalk server
   
2. It provides encryption
3. It provides a central configuration repository for multi server environment
   

Microsoft also provide a sample of how to use the SSO as configuration Store (http://go.microsoft.com/fwlink/?LinkId=99741)

How to configure our application configuration in SSODB
The SSO Installation comes some executable files in folder at C:\Program Files\Common Files\Enterprise Single Sign-On.
The executable which we're going to use is ssomanage.exe

As you can see in the screen shot above, there are some paramaters we can pass into the ssomanage.exe.

To create our custom application configuration :
1. Define our custom application configuration definition xml (below)
Note :
- When defining the fields, notice that the 1st field (ordinal=0) is a reserved field, so do not define any of your field in here, start defining our custom fields from the 2nd field (ordinal=1) afterwards.
- Use masked=true attribute to fields which need more security.

2. Save into xml file, e.g. BizAppTest1.xml

3. Run ssomanage.exe -createapps BizAppTest1.xml4. Go to C:\Program Files\Microsoft BizTalk Server 2006\SDK\Scenarios\Common\SSOApplicationConfig

5. Run Setup.bat to compile the tool, the executables should be generated in the bin folder (BTSScnSSOApplicationConfig.exe)

6. Use this tool to define our fields value :
Example :
BTSScnSSOApplicationConfig.exe -set BizAppTest1 "ConfigProperties" "ConnectionString" "Data Source=."

Note : Use "ConfigProperties" for the SSOIndentifierName parameter

7. Verify whether the value is correct
Example :
BTSScnSSOApplicationConfig.exe -get BizAppTest1 "ConfigProperties" "MaxInstances"

Richard Seroter has created his own awesome SSO Config Store Application Manager as window client tool to do the configuration in SSODB. You may want to take a look at it and try it yourself. He also has a lot of great biztalk posts and bits ;)

Note : If you have any additional fields or you want to remove the fields from SSO, the only way i know right now is to delete the app (ssomanage.exe -deleteapp BizAppTest1) and then re-do all the steps to setup the configuration. I created some script in a batch file so it will be much easier to do and can be re-used for deployment to production as well.

How to retrieve the configuration from SSODB
Fear not, when we install the enterprise single-sign on, there is an installed interface component which will provide the functionality to access the SSODB database, so we don't need to take care about how and where to connect to the SSODB database.

After you download the BizTalk Sample file for SSO As configuration store http://go.microsoft.com/fwlink/?LinkId=99741, there is a utility cs file, named SSOConfigHelper.cs. You can either include this class or the dll into your project.

Then we can retrieve our custom configuration value by calling this static method from the SSOConfigHelper class :

Microsoft.SSO.Utility.SSOConfigHelper.Read("BizAppTest1", "ConnectionString");

Hope this helps :)

BizTalk debugging with DebugView

First of all, happy new year 2008 :D

It's been a while since i did the the last blog.
But now since the current project is in the user testing phase, i guess i can post some articles about the interest things i found lately ;)

Since I moved to the new company, I have been working with BizTalk 2006, Oracle9i. So you will find more things about these 2 in my following posts ;)

Let's start with BizTalk debugging. One thing about developing biztalk application is that we will likely to spend more time in the testing rather than creating the orchestration. And the tricky part of it is that the debugging part is not as easy as we have in other .net application types.

Previously, I worked before with BizTalk 2004 about 1+ year ago, we used the file or event log to dump all the debugging output.
But now, life is much easier since i found DebugView. It will output all the trace and debug into the output window of the tool, which makes easier to debug/trace compared to the other methods (file - have to refresh/reopen the file, eventlog - have to move up and down to see each trace/debug item). And please note that this tool can be used for any type of applications, not only for BizTalk.

Below is the sample flow :
I put several debug / trace in the web application, BizTalk orchestration and the web services.
For example :

System.Diagnostics.Debug.WriteLine(varXMLSOAPHeader.OuterXml, "ConsumeWSOrchestration");

System.Diagnostics.Trace.WriteLine("Finish constructing WS message", "ConsumeWSOrchestration");

As you can see in the debugview screen shot below, it will output each debugging / tracing in the output window when the application is running.

How to decide whether using Debug or Trace?

• Debug : dump everything and as many as you want in the orchestration , such as message content, variables, etc.
• Trace : to monitor the orchestration flow, put the information as simple as possible. For example, we have a decide shape with 3 branches, i will use trace to inform me which branch the instance is actually going into.

Do we have to remove all the debug & tracing manually when deploying the orchestration to production? NO :

1. By default debug & trace will be enabled in the Debug compilation mode, and only trace will be enabled in the Release compilation mode. (you should be able to change this in the project build properties).
2. Make sure that you compile the DLLs with Release mode for production (so the debug symbols will not be included).

Design Consideration :
It will be better if instead of calling the System.Diagnostic class directly, you encapsulate the debug / tracing method with a helper class.

The benefit :

1. Maintainability
2. You can control the behaviour of debug / trace. E.g : add output to other channels such as file, event log

I have a helper class with a static method below which will handle the debug / trace logic

public static void Log(string appName, string orchestrationName, string body, LogType lt)
{
//LogToDebugView(appName, orchestrationName, body, lt);
//LogToEventLog(appName, orchestrationName, body, lt);
LogToFile(appName, orchestrationName, body, lt);
}

So i just need to call the method to debug / trace like this :

Logger.Log("appName", "orchestrationName", "This is a debug", Logger.LogType.Debug);

Normally, in production, we will use file / event log output not the debugview for administrative purposes.

Direct Link to download the DebugView :
http://download.sysinternals.com/Files/DebugView.zip

Overview of DebugView :
http://www.microsoft.com/technet/sysinternals/Miscellaneous/DebugView.mspx

No installation needed, just unzip the file and you are ready to go ;)

Note : Some of my friends got often hang issue with their PC when starting the debugview v4.73, after downloaded and using the newer version v4.74, they haven't encountered any hang issue since then and they are "attached" to the tool like I am :P

**Update : A nice feature of DebugView is Remote Monitoring. This is quite useful to monitor the applications when deployed in the DEV, SIT, UAT servers, so you don't even have to login to the server itself to see the dumps.