Showing posts with label Orchestration. Show all posts
Showing posts with label Orchestration. Show all posts

Friday, February 29, 2008

Application (Message) Based Routing in BizTalk Orchestration

This may not be a common scenario, but a few weeks ago, I was given this requirement :

If a message comes from Application 1, please send it to Web Service A in Server X, if it comes from Application 2, please send it to Web Service A in Server Y, so on and so forth.


Notes : Web Services A is a same webservices deployed on both Server X and Server Y

What are the purposes of having this requirement?

  1. Development :
    • We have numbers of projects which may be developed in different development servers but for the BizTalk parts, they are all deployed in a single BizTalk Development box. When we're calling the a particular web services, it is possible that Project A will use Development Server A and Project B will use Development Server B.
  2. Production :
    • For Load management purpose, we can point Application 1 calls to Web Server A and other Application calls to Web Server B.

In BizTalk, we can do it this way below, however since Web Service A in Server X and Server Y are actually a same web services, BizTalk will not be able to process the messages because they will have same namespace and same schema information. Unless you want to use different pipelines for each send port, imagine if you have 10 or more applications, do you really want to create additional custom pipelines for each of them :P

To solve the namespace and schema issue, we can create a common orchestration which can be shared / called by the applications. But wait, there's only 1 send port can be defined in this solution, we still want Application 1 to call Web Services A in Server X and Application 2 to call Web Services A in Server Y.


Now, this is where Role Link comes to play. If you see the diagram below, between the Orchestration and the send ports, there is a Role Link Provider. This is just like an additional routing layer which we can use instead of correlating the orchestration and the send port directly.


How to use Role Link is quite similar to regular Send port, you may notice when clicking on the port surface, there is New Role Link option ;) and you will need to choose between Provider & Consumer which is basically to define whether you will be sending message through this port or receiving message through this port.


The next question will be how to route the message?
  1. We will need to have a value in the message to differentiate where this message comes from and should be routed into
  2. We define the destination party before we send the message, e.g :
    varDestinationPartyName = HelperClass.GetDestinationPartyName(partyName);
    send_port1(Microsoft.XLANGs.BaseTypes.DestinationParty) = new Microsoft.XLANGs.BaseTypes.Party(
    varDestinationPartyName,"OrganizationName");
  3. We create new Party with the Organization name that we define, e.g : Application 1, Application 2, etc.
  4. After deployed the orchestration, we should be able to find the provider we defined in the orchestration in the Role Links folder under Orchestrations folder (BizTalk Administration Console)
  5. Click on the provider to open the the window. In this window, we can bind the party and the send ports. For illustration :
    • Party : Application 1, Logical Port : send_port1, Physical Port : sp_WSA_ServerX
    • Party : Application 2, Logical Port : send_port1, Physical Port : sp_WSA_ServerY
    • Party : Default, Logical Port : send_port1, Physical Port : sp_WSA_ServerY
Notes : I provide a default party as well, so in case if i have a new application 3 and i haven't added it to the party list, it will use the default party :)

How to check whether the party exists?

public static string GetDestinationPartyName(string partyName)
{
string result = Constant.DefaultDestinationPartyName;
BtsCatalogExplorer catalog = null;

try
{
// Create the root object and set the connection string
catalog = new BtsCatalogExplorer();
catalog.ConnectionString = ConfigurationHelper.BizTalkMgmtDBConnectionString;

// Get the requested party from the collection
Microsoft.BizTalk.ExplorerOM.Party party = catalog.Parties[applicationUserID];

if (party == null)
{
//Log as warning that the party could not be found
}
else
{
result = party.Name;
}
}
catch (Exception ex)
{
//Log the exception
}
finally
{
if (catalog != null)
catalog.Dispose();
}

return result;
}


Notes : Party List is not available at WMI

If you're getting this exception "Microsoft.BizTalk.ExplorerOM.BtsException: The database or the database version is incompatible with the installed version of this product.", you can just use sql objects to retrieve the party name from database instead of using the BizTalkCatalogExplorer :

cmd = new SqlCommand("select nvcName from bts_party where nvcName = @partyname", cn);
SqlParameter prmPartyName = new SqlParameter("@partyname", applicationUserID);
cmd.Parameters.Add(prmPartyName);


Hopefully, this interests you as it is for me ;)

Tuesday, February 26, 2008

Date Serialization Issue (GMT) in BizTalk Orchestration?

I just posted a question in the microsoft.public.biztalk.orchestration a few minutes ago.

I found Paul's post and Nigel's post in the net related about this date problem as well, someone mentioned that this is a .Net Serialization for DateTime. But the thing that i have yet to understand, why this only happens in BizTalk Orchestration? When the message is sent to the web services, it is also serialized right?

Nigel's solution is to handle the time offset difference in the Orchestration by ourselves, but is this a right way to do? What I'm afraid of is that if i follow this now, what if later in future, the date serialization is suddenly working fine (because of hotfix or whatever) then i find my self with bunch of wrong dates again :P

For the time being, we decided that we will pass the date in string format where we already have a current standard string format to follow. Why? Because currently this is not the only problem that we have, we're also having a different DateTime behavior with Oracle Adapter in Development and Production server aarrgghhh :(

I'll update this post with the solution when i find what, why, and how ;)
If you do have any existing solution, please let me know.

Problem :
Date becomes GMT in the BizTalk orchestration which cause wrong results (input <> output)

Scenario :
Web Application <-> Exposed Web Services By BizTalk Orchestration <-> BizTalk Orchestration <-> Consume Another Web Services

<-> : Request - Response communication.

BizTalk Orchestration will construct a message to consume another web services based on the request information from web application

Below message will be passed all the way in the process :
Note : DateInfo = xs:datetime, DateOnly = xs:date

Request Message :
<request xsi="http://www.w3.org/2001/XMLSchema-instance" xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://MyBizTalkApp.Response">
<message>Calling BizTalk Web Services at 2/26/2008 1:35:47 PM</message>
<dateinfo>2008-02-26T13:35:47.1828125+08:00</dateinfo>
<dateonly>2008-02-26</dateonly>
</request>

Web Service Message :
message : <string>Calling BizTalk Web Services at 2/26/2008 1:35:47 PM</string>
dateInfo : <datetime>2008-02-26T05:35:47.1828125Z</datetime>
dateOnly : <datetime>2008-02-25T16:00:00Z</datetime>

Trace Results :
WebApplication: DateInfo : 2008-02-26T13:35:47.1828125+08:00
WebApplication: DateOnly : 2008-02-26T13:35:47.1828125+08:00

ExposedWebServices: DateInfo : 2008-02-26T13:35:47.1828125+08:00
ExposedWebServices: DateOnly : 2008-02-26T00:00:00.0000000

BizTalkOrchestration: Request DateInfo : 2/26/2008 5:35:47 AM -> Date becomes GMT
BizTalkOrchestration: Request DateOnly : 2/25/2008 4:00:00 PM -> Date becomes GMT
BizTalkOrchestration: ConsumedWS Request DateInfo : 2008-02-26T05:35:47.1828125Z -> Get From Request DateInfo
BizTalkOrchestration: ConsumedWS Request DateOnly : 2008-02-25T16:00:00.0000000Z -> Get From Request DateOnly

ConsumedWebServices: DateInfo : 2008-02-26T05:35:47.1828125Z
ConsumedWebServices: DateOnly : 2008-02-25T16:00:00.0000000Z

Monday, February 4, 2008

Microsoft.XLANGs.Core.UnexpectedMessageTypeException: Received unexpected message type '' does not match expected type

Just drop a quick note on this.

If you ever find this BizTalk exception where you so sure that you didn't change anything and it just happens after you deploy your orchestration :
Check your pipeline settings in your send port, it might have changed into PassThruTransmit/PassThruReceive instead of XmlTransmit/XmlReceive.

Not sure why it's changed, but this sometime happens after re-deploying orchestration.

Another 2 hours wasted because of this :(

Friday, January 11, 2008

BizTalk Oracle Adapter / ADO.Net Limitation (ORA-01460: unimplemented or unreasonable conversion requested)

System defined exception
Call to RP_BIZTALKTEST@OracleDb://ORADEV/DBO/Procedures/TopLevel
Implementation = OracleDb://ORADEV/DBO/Procedures/TopLevel

Source: OracleDb
Error Code: 1460 (0x5b4)
Cause: OracleDb://exception=DBException (Unique ID )
42000 : [Oracle][ODBC][Ora]ORA-01460: unimplemented or unreasonable conversion requested


Exception data:
struct DBException =
WideString StatusCode = "42000"
WideString Description = "[Oracle][ODBC][Ora]ORA-01460: unimplemented or unreasonable conversion requested"


I'm using BizTalk 2006 with BizTalk Oracle Adapter to Oracle9i Database.

What i was trying to do here is to call an oracle procedure to insert a new row in a table and one of the field is CLOB/XMLType. This field is actually storing the current message which was passed in to the orchestration and to be used in future processing (other orchestration).

For simple / small size message, it was working fine as expected. Fortunately, I did try the scenario for huge incoming message (always test your code with the highest value of the assumptions (e.g. The user will be able to submit 1000 items in 1 transaction). Then I got this exception from BizTalk.

I tried to search through the net about this, it seems there is a limitation to pass in a parameter more than 32 KB size. (http://www.google.com/search?hl=en&q=32+kb+xmltype+parameter+procedure+limit&btnG=Search&meta=)

Some Tests which I did :
1. Use NativeSQL insert into table. - Not Working
XMLType column will be generated as base64binary type in the schema.
I have tried to pass the xml raw string but failed. Same if I tried to convert the xml into base64binary.
The error message which was returned :

System defined exception
Call to Insert@OracleDb://ORADEV/DBO/Tables/BIZTALKTEST
Implementation = OracleDb://ORADEV/DBO/Tables/BIZTALKTEST

Source: OracleDb
Error Code: 932 (0x3a4)
Cause: OracleDb://exception=DBException (Unique ID )
HY000 : [Oracle][ODBC][Ora]ORA-00932: inconsistent datatypes: expected NUMBER got BINARY


Exception data:
struct DBException =
WideString StatusCode = "HY000"
WideString Description = "[Oracle][ODBC][Ora]ORA-00932: inconsistent datatypes: expected NUMBER got BINARY
"


Or (I tried some combinations for the data type : CLOB/XMLTYPE)

Error transmitting message: SOAP-ENV:ServerRequest ID: Unknown
Exception Type: A cross object system conversion error occurred
Exception Info:
Exception occurred:
E-CORE0053: Dimensions or bounds of source and target params do not match
Call to Insert@OracleDb://ORADEV/DBO/Tables/BIZTALKTEST
Implementation = OracleDb://ORADEV/DBO/Tables/BIZTALKTEST

SOAP://xmlns/schemas.microsoft.com/BIZTALKTEST/InsertRecord:Type/[]/TESTRAW ==> OracleDb://ORADEV/DBO/Tables/BIZTALKTEST/InsertRecord/VARCHAR2/WideString
(SOAP://xmlns/schemas.microsoft.com/BIZTALKTEST/Insert:Element/Rows ==> sequence<struct OracleDb://ORADEV/DBO/Tables/BIZTALKTEST/InsertRecord&>)
(Error occurred before call to foreign method)


2. Call oracle Procedure passing XMLTYPE parameter - Not Working
I tried to use XMLType as a parameter but there was an error similar to the 2nd error above.

Error transmitting message: SOAP-ENV:ServerRequest ID: Unknown
Exception Type: A cross object system conversion error occurred
Exception Info:
Exception occurred:
E-CORE0053: Dimensions or bounds of source and target params do not match
Call to RP_BIZTALKTEST@OracleDb://ORADEV/DBO/Procedures/TopLevel
Implementation = OracleDb://ORADEV/DBO/Procedures/TopLevel

SOAP://xmlns/schemas.microsoft.com/TopLevel/RP_BIZTALKTEST:Element/V_TESTRAW ==> OracleDb://ORADEV/UNDEFINED_2000/WideString
(SOAP://xmlns/schemas.microsoft.com/TopLevel/RP_BIZTALKTEST:Element/V_TESTRAW ==> OracleDb://ORADEV/UNDEFINED_2000)
(Error occurred before call to foreign method)


3. Call oracle Procedure passing CLOB parameter and then convert into XMLTYPE when insert - Not Working
This is the one that actually has 32KB limitation. I actually created a small .net application using oracle client to test this and turned out that I got the same error when i tried to pass > 32 KB string into the parameter. There is a work around in this article, but i find it more complex than the 4th solution below (http://msdn2.microsoft.com/en-us/library/cydxhzhz(VS.80).aspx)

System defined exception
Call to RP_BIZTALKTEST@OracleDb://ORADEV/DBO/Procedures/TopLevel
Implementation = OracleDb://ORADEV/DBO/Procedures/TopLevel

Source: OracleDb
Error Code: 1460 (0x5b4)
Cause: OracleDb://exception=DBException (Unique ID )
42000 : [Oracle][ODBC][Ora]ORA-01460: unimplemented or unreasonable conversion requested


Exception data:
struct DBException =
WideString StatusCode = "42000"
WideString Description = "[Oracle][ODBC][Ora]ORA-01460: unimplemented or unreasonable conversion requested"


4. Use Oracle Data Provider for .NET 9.2.0.7.00 (ODP.Net) - Working
I found that this is actually the solution and the easiest way to do, passing a XMLTYPE is just as simple as passing other parameters, It has its OracleDb Type.XmlType. So far I have tested with huge messages more than 32 KB and they are working fine. Please also note on the ODP.Net version.

OracleParameter pContent = new OracleParameter("content", OracleDbType.XmlType, content.Length);

Before I decided to go with the ODP.Net, I have tried using :
1. ODBC Connection (System.Data.Odbc) : Limited with 4 KB size
2. .Net Oracle Data Provider (System.Data.OracleClient) : Limited with 32 KB size as well.

Thursday, January 10, 2008

BizTalk Consume WebServices with SOAP Header

When working with Web Services, the common scenario is to pass in the credential in the SOAP Header.

In this post, I will provide the steps how to do this in BizTalk Orchestration.
This will be a simple flow like below screen shot :

Flow Detail :

  1. Web Application will call an exposed web services by Biztalk Orchestration
  2. The request object consists of 3 elements : UserName, Password, Message
  3. BizTalk Orchestration will construct the SOAP Header Message with UserName and Password
  4. BizTalk Orchestration will assign the Message as the body and assign the SOAP Header
  5. BizTalk Orchestration calls the Web Services (WSSOAPHeader)
  6. Web Services will read the SOAP Header message and return back some message
  7. BizTalk will also return back the message to the web application

BizTalk Orchestration layout

One of the most crucial thing in here is the SOAP Header message.
To create the SOAP Header message schema :
  1. Create a property schema file
  2. In the schema properties, change the Target Namespace to http://schemas.microsoft.com/BizTalk/2003/SOAPHeader Set the Node Name property to the SOAP Header Name and the Property Schema Base to MessageContextPropertyBase
After creating this property, we can assign the SOAP Header into the message like this below

varXMLSOAPHeader = msgSOAPHeader;
msgWSRequest(MyBizTalkApp.AuthenticationSOAPHeader) = varXMLSOAPHeader.OuterXml;


Note :
  • When assigning the SOAP Header, it expects string value, that's why I assign the msgSOAPHeader into an XML document variable and get the string from OuterXml property.
  • If the SOAP Header is not received by the web services even though you have assigned it in the code, make sure that you have followed each of the steps to create the SOAP Header message schema above, especially the namespace part. It caused me 2 days to notice this one in the end :P
You can find and download the whole VS 2005 solution at here :


Here is the link to the MSDN Site : Consuming Web Services with SOAP Headers

Hope this helps

Friday, January 4, 2008

SSODB Configuration Store for BizTalk Orchestration

One of the aspect of the application design is the configuration module.

There are at least 4 repositories where we can store the configuration for BizTalk orchestration :

  1. Custom config file
  2. BizTalk config file
  3. Windows Registry
  4. Database
Since each company will have its own policy for the production environment, this will eventually lead into the configuration repository.

Each repository above has its own pros and cons to implement, but I would say that configuration files and registry will not scale for MultiServer / Clustering environment, since we need to replicate them to each of the BizTalk server. So it will be easier to use database since we can put the configuration information in one place and all the servers will access this central database.

But when it comes to the database, we will have the next question of where to store the connection string. Well, this is the point where SSODB comes in handy.

As we may notice, whenever we install BizTalk Server, the Enterprise Single Sign-on will need to be installed as well, and the Enterprise Single Sign-on has a database, named SSODB. This is mainly used for BizTalk to store all the internal configuration information.

This SSODB will be the ideal solution for the configuration repository since as far as i know.
Benefits :
  1. SSO will be installed and running along with BizTalk server
  2. It provides encryption
  3. It provides a central configuration repository for multi server environment
Microsoft also provide a sample of how to use the SSO as configuration Store (http://go.microsoft.com/fwlink/?LinkId=99741)

How to configure our application configuration in SSODB
The SSO Installation comes some executable files in folder at C:\Program Files\Common Files\Enterprise Single Sign-On.
The executable which we're going to use is ssomanage.exe

As you can see in the screen shot above, there are some paramaters we can pass into the ssomanage.exe.

To create our custom application configuration :
1. Define our custom application configuration definition xml (below)
Note :
- When defining the fields, notice that the 1st field (ordinal=0) is a reserved field, so do not define any of your field in here, start defining our custom fields from the 2nd field (ordinal=1) afterwards.
- Use masked=true attribute to fields which need more security.

2. Save into xml file, e.g. BizAppTest1.xml

3. Run ssomanage.exe -createapps BizAppTest1.xml4. Go to C:\Program Files\Microsoft BizTalk Server 2006\SDK\Scenarios\Common\SSOApplicationConfig

5. Run Setup.bat to compile the tool, the executables should be generated in the bin folder (BTSScnSSOApplicationConfig.exe)

6. Use this tool to define our fields value :
Example :
BTSScnSSOApplicationConfig.exe -set BizAppTest1 "ConfigProperties" "ConnectionString" "Data Source=."

Note : Use "ConfigProperties" for the SSOIndentifierName parameter

7. Verify whether the value is correct
Example :
BTSScnSSOApplicationConfig.exe -get BizAppTest1 "ConfigProperties" "MaxInstances"

Richard Seroter has created his own awesome SSO Config Store Application Manager as window client tool to do the configuration in SSODB. You may want to take a look at it and try it yourself. He also has a lot of great biztalk posts and bits ;)

Note : If you have any additional fields or you want to remove the fields from SSO, the only way i know right now is to delete the app (ssomanage.exe -deleteapp BizAppTest1) and then re-do all the steps to setup the configuration. I created some script in a batch file so it will be much easier to do and can be re-used for deployment to production as well.

How to retrieve the configuration from SSODB
Fear not, when we install the enterprise single-sign on, there is an installed interface component which will provide the functionality to access the SSODB database, so we don't need to take care about how and where to connect to the SSODB database.

After you download the BizTalk Sample file for SSO As configuration store http://go.microsoft.com/fwlink/?LinkId=99741, there is a utility cs file, named SSOConfigHelper.cs. You can either include this class or the dll into your project.

Then we can retrieve our custom configuration value by calling this static method from the SSOConfigHelper class :

Microsoft.SSO.Utility.SSOConfigHelper.Read("BizAppTest1", "ConnectionString");


Hope this helps :)

Wednesday, January 2, 2008

BizTalk debugging with DebugView

First of all, happy new year 2008 :D

It's been a while since i did the the last blog.
But now since the current project is in the user testing phase, i guess i can post some articles about the interest things i found lately ;)

Since I moved to the new company, I have been working with BizTalk 2006, Oracle9i. So you will find more things about these 2 in my following posts ;)

Let's start with BizTalk debugging. One thing about developing biztalk application is that we will likely to spend more time in the testing rather than creating the orchestration. And the tricky part of it is that the debugging part is not as easy as we have in other .net application types.

Previously, I worked before with BizTalk 2004 about 1+ year ago, we used the file or event log to dump all the debugging output.
But now, life is much easier since i found DebugView. It will output all the trace and debug into the output window of the tool, which makes easier to debug/trace compared to the other methods (file - have to refresh/reopen the file, eventlog - have to move up and down to see each trace/debug item). And please note that this tool can be used for any type of applications, not only for BizTalk.

Below is the sample flow :
I put several debug / trace in the web application, BizTalk orchestration and the web services.
For example :

System.Diagnostics.Debug.WriteLine(varXMLSOAPHeader.OuterXml, "ConsumeWSOrchestration");

System.Diagnostics.Trace.WriteLine("Finish constructing WS message", "ConsumeWSOrchestration");


As you can see in the debugview screen shot below, it will output each debugging / tracing in the output window when the application is running.

How to decide whether using Debug or Trace?

  • Debug : dump everything and as many as you want in the orchestration , such as message content, variables, etc.
  • Trace : to monitor the orchestration flow, put the information as simple as possible. For example, we have a decide shape with 3 branches, i will use trace to inform me which branch the instance is actually going into.
Do we have to remove all the debug & tracing manually when deploying the orchestration to production? NO :
  1. By default debug & trace will be enabled in the Debug compilation mode, and only trace will be enabled in the Release compilation mode. (you should be able to change this in the project build properties).
  2. Make sure that you compile the DLLs with Release mode for production (so the debug symbols will not be included).
Design Consideration :
It will be better if instead of calling the System.Diagnostic class directly, you encapsulate the debug / tracing method with a helper class.

The benefit :
  1. Maintainability
  2. You can control the behaviour of debug / trace. E.g : add output to other channels such as file, event log
I have a helper class with a static method below which will handle the debug / trace logic

public static void Log(string appName, string orchestrationName, string body, LogType lt)
{
//LogToDebugView(appName, orchestrationName, body, lt);
//LogToEventLog(appName, orchestrationName, body, lt);
LogToFile(appName, orchestrationName, body, lt);
}

So i just need to call the method to debug / trace like this :

Logger.Log("appName", "orchestrationName", "This is a debug", Logger.LogType.Debug);

Normally, in production, we will use file / event log output not the debugview for administrative purposes.

Direct Link to download the DebugView :
http://download.sysinternals.com/Files/DebugView.zip

Overview of DebugView :
http://www.microsoft.com/technet/sysinternals/Miscellaneous/DebugView.mspx

No installation needed, just unzip the file and you are ready to go ;)

Note : Some of my friends got often hang issue with their PC when starting the debugview v4.73, after downloaded and using the newer version v4.74, they haven't encountered any hang issue since then and they are "attached" to the tool like I am :P

**Update : A nice feature of DebugView is Remote Monitoring. This is quite useful to monitor the applications when deployed in the DEV, SIT, UAT servers, so you don't even have to login to the server itself to see the dumps.