BizTalk Servers Slow First Hit

2 08 2010

Lately I been thinking about BizTalk Server, and a particular behavior that it consistently demonstrates without fail. It takes a dreadful amount of time to service a “cold” request, however once “warmed”, it hums.

Its challenging at best to justify this behavior to a technology ignorant client.

Without getting too deep into BizTalk Servers internals (I would love to spend some time with windbg and the SOS extension digging around), I wanted a way to have partial control over how BizTalk Server manages the actual processes that invoke the code we make. All mainstream BizTalk artefacts (orchestrations, maps, pipelines) boil down to managed code (IL). BizTalk consumes our crafted “business” assemblies (dll’s) by loaded them into its address space through one or more AppDomain’s, at which time the messaging engine can call out to them when it sees fit.

This spawns a number of related questions; how many dll’s per appdomain? Under what conditions does an appdomain’s get “garbage collected” by the messaging engine? And so on. So I digged a little deeper.

Thanks to Tomas Restrepo for posting the excellent MSDN link to Orchestration Engine Configuration, which in essence sums up everything I wished for. Basically it involves hacking the BTSNTSvc.exe.config, which host instances take into account when started. While you can do cool things like control dehydration behaviour, and more, I was more interested in this:

Assemblies are assigned to named domains using assignment rules (see more below). If no rule is specified for some assembly, the assembly will be assigned to an ad hoc domain. The number of such assigned assemblies per ad hoc domain is determined by the value of AssembliesPerDomain.

Which translates to this in btsntsvc.exe.config:

<AppDomains AssembliesPerDomain="10">
      <AppDomainSpec Name="FooDomain" SecondsIdleBeforeShutdown="-1" SecondsEmptyBeforeShutdown="-1" />
      <ExactAssignmentRule AssemblyName="Foo.Orchestration, Version=, Culture=neutral, PublicKeyToken=9f3a0f87e62e465c" AppDomainName="FooDomain" />

The interesting properties SecondsEmptyBeforeShutdown and SecondsIdleBeforeShutdown, are defined as follows:

SecondsEmptyBeforeShutdown is the number of seconds that an app domain is empty (that is, it does not contain any orchestrations) before being unloaded. Specify -1 to signal that an app domain should never unload, even when empty.

SecondsIdleBeforeShutdown is the number of seconds that an app domain is idle (that is, it contains only dehydratable orchestrations) before being unloaded. Specify -1 to signal that an app domain should never unload when idle but not empty. When an idle but non-empty domain is shut down, all of the contained instances are dehydrated first.

Thanks to Mick Badran, who has posted a handy BTSNTSvc.exe.config template, which includes the AppDomain configuration section discussed above.

UPDATE: Here’s a copy of my own *no frills* template:

<?xml version="1.0" ?>
    <section name="xlangs" type="Microsoft.XLANGs.BizTalk.CrossProcess.XmlSerializationConfigurationSectionHandler, Microsoft.XLANGs.BizTalk.CrossProcess" />
      <add address="*" maxconnection="48"/>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <probing privatePath="BizTalk Assemblies;Developer Tools;Tracking;Tracking\interop" />
        <provider id="sspi" type="Microsoft.BizTalk.XLANGs.BTXEngine.SecurityServerChannelSinkProvider,Microsoft.XLANGs.BizTalk.Engine" securityPackage="ntlm" authenticationLevel="packetPrivacy" />
        <channel ref="tcp" port="0" name="">
            <provider ref="sspi" />
            <formatter ref="binary" typeFilterLevel="Full"/>
      <AppDomains AssembliesPerDomain="10">
          <AppDomainSpec Name="FooDomain" SecondsIdleBeforeShutdown="-1" SecondsEmptyBeforeShutdown="-1" />
          <AppDomainSpec Name="BarDomain" SecondsIdleBeforeShutdown="-1" SecondsEmptyBeforeShutdown="-1" />
          <PatternAssignmentRule AssemblyNamePattern="Net.benCode.Foo.*, Version=\d.\d.\d.\d, Culture=neutral, PublicKeyToken=.{16}" AppDomainName="FooDomain" />
          <PatternAssignmentRule AssemblyNamePattern="Net.benCode.Bar.*, Version=\d.\d.\d.\d, Culture=neutral, PublicKeyToken=.{16}" AppDomainName="BarDomain" />

Performance Analysis

28 01 2008

Log Parser: An elegant utility that does all the real grunt work. It is well worth spending some time with this guy as it will come in handy for other unexpected situations, like analysing your web server logs, or parsing through custom CSV files etc.

Microsoft Office 2003 Web Components: Some COM components that provide for graph generation of the statistical data.

Performance Analysis of Logs: A script that automates the “leg work” involved in real-world performance analysis, such as managing the set of counters to be used for particular situations (eg. BizTalk 2006 analysis), invocation of log parser to do the actual analysis of each counter, detection of threshold breaches when things seem to be fishy. PAL is implemented as a VBScript, and comes with a little .NET WinForm GUI which can be (optionally) used to setup the arguments to be fed into the VBScript. The result, a comprehensive HTML report complete with alerts (threshold breaches), graphs and explanations.

PAL analyses log files, it does not collect them—although there is talk of a future version supporting this. However, a suite of helper scripts (again VBScript) ship with PAL 1.1.7 and higher. These are tucked away in the %PAL_Directory%\PerfmonLogScripts directory. Here’s what I did for my BizTalk test box—invocation to remote servers is supported.

1. cscript CreateAndStartPerfmonLogs.vbs MyTestBox BizTalk CounterList_BizTalk2006.txt

2. Stop the performance counter collector set when done. There is a script to help do this, but I found it easier to stop the collector set using perfmon. Fire up Perfmon > Data Collector Sets > User Defined > HealthCheck_BizTalk_MyTestBox > Click the stop button.
There should be a performance log that corresponds to the collection just ran in c:\perflogs.

3. Start up the PAL (PAL.exe) front-end to configure its arguments. Point it to the newly created performance log file, and make sure that the threshold file drop-down corresponds to the counters which you collected (eg. its not as useful to use the “System Overview” threshold template when you have the performance data for “BizTalk Server 2006”). The rest of the form is self-explanatory.
After a period of time (depending how much data has been collected) an HTML report should present itself.