Tim Rayburn's Blog - EDI - Connected People make Connected Systems

Great news!  When I attended and spoke at the Microsoft SOA & Business Process Conference I was informed that they were recording all sessions to be included in a DVD of the event which would be sent out to those who had attended.  Great, wonderful, but not a lot of use to my blog readers.  Well shortly there after I received a request from the TechNet team to include my presentation as part of the "Best Of" from event online.  I happily agreed, and am happy to report that the best of the Microsoft SOA & Business Process Conference can now be found on TechNet, and that my presentation on Building HIPAA solutions with BizTalk Server 2006 R2 is amongst them.  Incidentally apparently I received a 5 of 5 rating during the conference, I thank the attendees for being so easy on me.

This talk is actually very relevant to anyone interested in EDI for BizTalk Server 2006 R2 as the HIPAA specific pieces are few.  It is a good introductory talk, it is not a deep internals talk but it does cover the configuration and setup of EDI/HIPAA in R2 in some depth.  If you're interested in this topic and can't get to an event where I'm speaking, this is a nice alternative.

I'm quite certain I won't be the only person who tries this, I had a need to send an encoded EDI message to a Web Service, so I was like "No problem, we're in the high tech land of BizTalk 2006 R2 now, I'll just call a Pipeline directly from an Orchestration, get the results and send that along."

Not so quick, buck-o replied BizTalk, and after some Googleing I came up with this link to a post on the Microsoft forums which informed me that you can't do that.  Specifically, to quote it here in case that forum thread disappears, the question was "Steven G." and asked (in part) :

I have an orchestration which is trying to send EDI files using a dynamic send port.

I have been trying, with no success, to call the "Microsoft.BizTalk.Edi.DefaultPipelines.EdiSend" from within my orchestration, as follows:

And the response can back from Tony Bernard of Microsoft:

Running the EDISend pipeline from an orchestration is not supported in R2.  We are tracking this for the future, but had higher priority items to address for this release.

I would suggest one of two approaches:

• a post-processing pipeline component which performs the manipulation that you need to occur after serialization

• using a loopback port. Basically, use a send port with that pipeline into an MSMQ or file location and a receive location with a pass through pipeline back into the orchestration

So no love there.  I ended up working around the problem, but know I will have to solve this again in a few days, so be looking for a post from me on how to implement the Loopback port which Tony suggested.

Today I was taught an immensely important lesson by WCF.  It actually has nothing to do with WCF except that because of the size of the schema I'm working with that is where I encountered it.

I do a lot of work in the healthcare industry, and that industry, by government mandate, works with a message format called the 837 (properly the X12 4010A1 837 either I, P or D).  The 837 has three flavors, and is absolutely huge.  I was crafting a service which had to accept an XML representation of this data. How did I create the XML?  BizTalk Server 2006 R2's new HIPAA support of course.  When I made the call to service from my unit tests I first received a Fault indicating that exception details could not be returned because of configuration.  No problem, obviously I've seen this before and so I added the following to my app.Config for the Host executable:

<behaviors>

  <serviceBehaviors>

    <behavior name="EnableDebugging">

      <serviceDebug includeExceptionDetailInFaults="true"/>

    </behavior>

  </serviceBehaviors>

</behaviors>

Now that I've enabled debugging, surely now I'll see some sort of clear reason by this occurred. Nope!  I was presented with the following:

TheCompany.Services.Tests.ClaimsService.GetListOfClaimsByStatus : System.ServiceModel.FaultException`1[System.ServiceModel.ExceptionDetail] : Error in deserializing body of request message for operation 'GetClaimListByStatus'.

Well isn't that terribly useful.  Fortunately, this did give me the single most important piece of information "deserializing".  So off I go to Google my brain, because I'd written a post before on how to debug the XmlSerializer (and I was using the XmlSerializer in this case because the DataContractSerializer could not work with my schema).  As that post will tell you, there is a switch that can be set in the app.Config which will enable you to debug the code generated by the XmlSerializer.  So now we add the following to my app.Config:

<system.diagnostics>

  <switches>

    <add name="XmlSerialization.Compilation" value="1" />

  </switches>

</system.diagnostics>

And we follow that by running the unit tests again, that results in something that is actively useful:

For those reading along at home and wanting the full text that says:

The maximum nametable character count quota (10000) has been exceeded while reading XML data. The nametable is a data structure used to store strings encountered during XML processing - long XML documents with non-repeating element names, attribute names and attribute values may trigger this quota. This quota may be increased by changing the MaxNameTableCharCount property on the XmlDictionaryReaderQuotas object used when creating the XML reader.

Now that is useful, it tells me exactly how to correct the problem and the size of the quota currently.  The only problem is that I didn't create this XmlReader object, I didn't even create the XmlSerializer which created this XmlReader.  The only thing which I create was a ServiceHost, how can I change that setting?  Enter Google again.

As it happens, Windows Communication Foundation does include an ability to override that value, it is stored in the Binding Configuration in your ... you guessed it ... app.Config!  Add the following snippet (assuming you're using Basic Http Binding (it exists for all bindings though) and you'll be good to go:

<bindings>

  <basicHttpBinding>

    <binding name="BasicHttpBinding">

      <readerQuotas maxNameTableCharCount="100000"/>

    </binding>

  </basicHttpBinding>

</bindings>

Now, surely I'm done, and things will work.  Actually, no it took me a few times to find a value that would work for that quota.  Once I did though I finally hit my breakpoint inside the service and was able to send back a response, where upon I got this message:

TheCompany.Services.Tests.ClaimsService.GetListOfClaimsByStatus : System.ServiceModel.CommunicationException : Error in deserializing body of reply message for operation 'GetClaimListByStatus'.
  ----> System.InvalidOperationException : There is an error in XML document (1, 252).
  ----> System.Xml.XmlException : The maximum nametable character count quota (10000) has been exceeded while reading XML data. The nametable is a data structure used to store strings encountered during XML processing - long XML documents with non-repeating element names, attribute names and attribute values may trigger this quota. This quota may be increased by changing the MaxNameTableCharCount property on the XmlDictionaryReaderQuotas object used when creating the XML reader. Line 1, position 252.

At least when the error occurs on the client side you get the full details to start with!  Make the same modification above to your configuration on the client side and you're good to go.

Thanks to Tomas Restrepo for pointing out that the Developer Edition of BizTalk 2006 R2 is now up on MSDN.  If you're working with EDI and BizTalk server you seriously need to check this out and realize what is coming your way.

I'm creating a new series of posts related to using the BizTalk Mapper.  I know, it's certainly less sexy than Pipeline Testing Framework (seriously, follow that link) or many other BizTalk subjects, but so much of the magic of BizTalk happens in the mapping.

In my experience, when people are introduced to the BizTalk Mapper they get it very quickly.  The user interface is straight forward for most of what you need to do in mapping output.  But certainly not all.  Take for example some data that looks like this :

<?xml version="1.0" encoding="utf-8" ?>

<MyRoot>

  <ref qualifier="D2" value="999999999" />

  <ref qualifier="XX" value="999999999" />

  <ref qualifier="ZZ" value="XX-YYYYYYYY" />

  <nm qualifier="NI" value="999999999" />

</MyRoot>

Your mission, should you choose to accept it is simple.  Output one and only one value to destination schema in a node called Identifier which contains either the value of the 'value' attribute of the 'nm' element if the matching qualifier is 'ZZ' or if that does not exist the 'value' attribute of any 'ref' element containing a 'qualifier' attribute of 'ZZ'.  If neither exists, then output the constant value 'N/A'.

The catch?  Both nodes can both exist with the proper qualifier, and 'nm' must take precedence in that case.

Now, I've no doubt at all that someone could come up with some combination of functoids which might solve this problem.  But we must remember that in the end a map is just XSLT and this problem is supremely easy in XSLT and extremely difficult with the mapper.  The solution, as you've likely already guessed, is an XSLT Call Template Scripting Functoid.

An XSLT Call Template Scripting Functoid lets you inject your own XSLT into the middle of a map.  No need to take the effort of writing the full XSLT yourself if only one or two mappings are giving you trouble.  Obviously to be able to write such a functoid you need to understand XSLT, which if you are mapping is a good skill to have anyway.  In this simple example we don't even have to deal with parameters.  By entering a template as follows, you can solve this problem in a few short lines.

  <xsl:template name="MapIdentifier">

    <xsl:variable name="nmValue" select="/MyRoot/nm[@qualifier='ZZ']/@value" />

    <xsl:variable name="refValue" select="/MyRoot/ref[@qualifier='ZZ']/@value" />

    <xsl:element name="Identifier">

      <xsl:choose>

        <xsl:when test="$nmValue">

          <xsl:value-of select="$nmValue"/>

        </xsl:when>

        <xsl:when test="$refValue">

          <xsl:value-of select="$refValue"/>

        </xsl:when>

        <xsl:otherwise>N/A</xsl:otherwise>

      </xsl:choose>

    </xsl:element>

  </xsl:template>

Some things to remember when working with XSLT Call Templates.

1. Only use these when needed.  I recently heard Roy Osherove give the same advice regarding Regular Expression and he was right.  Both RegEx and XSLT Call Templates are more cryptic and harder for someone coming after you to decipher.  Use them when you must, when they save you time, but don't lean on them for everything.
2. The Template name must be unique within the map.  If you copy-paste the template and don't change the name then only one version will be run.  If you later modify one of them and start going nuts trying to figure out why the second isn't behaving properly, check your names.
3. XSLT Call Templates must create the destination node.  Element or Attribute they don't care, when you write one of these you are taking on the job of outputting that node, namespace and all. 

I hope this has been a good quick primer on using XSLT Call Templates.  If you've got a topic you'd like covered, leave a comment!

[Post Script : Before anyone comments, the example XML given was deliberately rendered similar to an EDI structure because this is a common problem in EDI mappings.]

So you need to handle an X12 EDI message and likely store it's data back into your database. If this is your first time trying to do such a thing, you are probably looking at a structure something like this:

ISA*00*          *00*          *ZZ*AV09311993     *01*030240928      *031023*1758*U*00401*557988899*1*T*:~
GS*HS*AV01101957*030240928*20031023*17581205*1*X*004010X092A1~
ST*270*0001~
BHT*0022*13*19637886*20031023*17581205~
HL*1**20*1~
NM1*PR*2*HUMANA*****PI*HUMANA~
HL*2*1*21*1~
NM1*1P*2*Hospital Name*****FI*111111111~
HL*3*2*22*0~
TRN*1*19637886*3030240928~
NM1*IL*1*LastName*FirstName****MI*1111111111~DMG*D8*20000101*F~
DTP*472*D8*20031023~
EQ*30~SE*13*0001~
GE*1*1~
IEA*1*557988899~

Note : The word wrapping has been done for readability, a properly formatted message would not have random CRLFs littered about, as is discussed below.

And your wondering what sadist created this standard, and how to handle it. First, let me tell you ... it gets worse before it gets better. If you intend to do alot of parsing of EDI, seriously look into BizTalk server (possibly with the EDI expansion offered by Covast) or one of its competitors. But EDI is easy to work with, once you understand the standards by which it is created, so I highly recommend going out to http://www.x12.org and finding the specifications for the type of EDI document you are working with.

An EDI document is just a delimited test file so the first thing to understand is that EDI has three different delimiters at work in it.

1. Field Delimiters (* in the example above)
2. Subfield Delimiters (: in the example above)
3. Record Delimiters (~ in the example above)

The X12 standard does not set down any rules which state that these delimiters must be certain values. * and ~ for record and field are very common but you can't assume that everyone will follow this convention. So if the standard does not set down what the delimiters are how can you parse this delimited file, you ask?

Well the folks who set down the X12 EDI standards were bright folks, and while EDI as a whole is a delimited, the first record in every file (ISA) is positional as well as delimited. Every field in the ISA record has a set length it must be, and as such you can find data in this first record either by using the delimiters, or by searching in a certain number of characters. Now, positional files are more of a bear to deal with in most modern programming languages, so we are going to do as little postional parsing as possible. We just need to get our delimiters, and then we can set about parsing the data in a delimited fashion.

According to the X12 specification, the very last field in the ISA record has a value equal to the subfield delimiter for that file. This field is located at position 105 within the file, and did I mention it was the last field in the record? This means that position 106 will be the equal to the record delimiter within your file and position 104 is equal to the field delimiter (because every field of data has the delimiter between it and the next, even this one). So to recap we can find our delimiters at:

1. Field Delimiter - Position 104
2. Subfield Delimiter - Position 105
3. Record Delimiter - Position 106

From this point on, it should be easy sailing. If you are working in a language like Visual Basic you can use the Split( ) function to take the entire file and split it by the record delimiters. Them you can take those records and split them by the Field Delimiters and finally if the field contains a Subfield delimiter you can split by that.

A couple of final pieces of advice as you begin to walk down this road:

• Build error check into your code ... trust me, you'll get badly formed messages and it will save you alot of time.
• The X12 standard sets down that every record has a tag associated with is (ISA, GS, ST, etc) which starts the record. When you split your data, be sure to check if the first character of the line is an alpha character. If not, you'll want to trim out any white space characters before continueing. The most common offending character that slips in is the line feed (LF) character. This happens when someone decides to take the more readable path of making a 'return' the record delimiter (which puts each record on its own line, which is nice) but doesn't realize that on their system 'return' is CRLF, hence two characters and a technical violation of the standard. You won't get people to every realize this, so just code for it, trust me.
• Start your own EDI parsing library, trust me it will come in handy.
• Seriously look at an Enterprise Application Integration (EAI) package like BizTalk or GEIS or the like if your volume gets very high. They are costly, but once you start getting into parsing the particulars of every different type of message (997, 824, 850, 837I, 837P, 837D, etc) you will begin to understand why such systems have been developed.