@michaelLipscombe 

Entries in ShortCut (4)

Friday
Apr012011

Basic FIX Routing Application using ShortCut

This is the fourth post in a short series on the FIX adapter for BizTalk.

  1. FIX for BizTalk (The basics)
  2. ShortCut Architecture and Overview
  3. Installing and Configuring ShortCut
  4. Basic Routing Application using RapidAddition’s ShortCut
  5. Mapping FIX Messages
  6. More complicated scenarios and patterns using a FIX-enabled BizTalk

This post will take you through a basic routing application in BizTalk using RapidAddition’s ShortCut FIX adapter. We’ll leverage content based routing (CBR) and although an unlikely easy scenario it will demonstrate how to use FIX messages in BizTalk.

The solution will focus on three entities.

BUYSIDEFIRM

This entity represents a buy side firm that will be sending orders to our message broker and we’ll route them to the appropriate hub or sell side firm. Our system will first ack back to the buy side, forward the messages on and then route the executions (sell side responses) back to the BUYSIDEFIRM. We will always ack the buyside but this is where some business process and perhaps the rules engine might come in handy to validate the orders.

FIX Details we’ll use for the receive location (from their perspective):

Property Value Description
IPAddress 127.0.0.1 This will all run on a single host
Port 9001  
SenderCompId BUYSIDEFIRM Counterparties identifier
TargetCompId MYMSGBROKER this is our BizTalk group
Initiator False BizTalk will initiate this session
Version FIX.4.2 The version of FIX that the buyside will support
HeartBeatInterval 60 seconds The interval that heartbeats will be sent, it’s important these match otherwise the sessions will logout.

 

In BizTalk we’ll need to configure a receive location and one send port for the BUYSIDEFIRM. The receive location will receive any messages the BUYSIDEFIRM sends over and the send port will send back our ack and then the fills we’ll get from the SELLSIDEFIRM. This will be apparent later and I’ll post a video of the solution in action which will help. 

The receive location, which will be from BizTalk’s perspective will be configured as:

image

Property Value Description
CounterParty: CompID BUYSIDEFIRM Our counterparties CompID (TargetCompId to BizTalk SenderCompID from their perspective)
IPAddress 127.0.0.1  
Port 9001  
FIXVersion FIX.4.2  
Initiator True BizTalk will initiate the session
HeartbeatInterval 60  
Party: CompID MYMSGBROKER  

 

For reasons discussed in the first article the send port is pretty straight forward.

image 

Property Value Description
CounterParty: CompID BUYSIDEFIRM Our counterparties CompID (TargetCompId to BizTalk SenderCompID from their perspective)
FIXVersion FIX.4.2  
Party: CompID MYMSGBROKER  

 

image

We want the BUYSIDE FIRM send port to subscribe to messages that come back from the SELLSIDE’s session. To construct the initial ack we’ll also subscribe to the BUYSIDE receive port and map the Order message to a ExecutionReport. 

image

Using the sweet new 2010 feature of only showing relevant nodes the map is pretty simple.

image

To represent the BUYSIDEFIRM we’ll use the FIX test harness that RapidAddition provide with their adapter. By now you should be getting the gist of what we’re trying to do.

image

This test harness will listen for logon messages over port 9001, if the logon message has the expected CompID’s the session will be established. If you never sent an order over the two FIX engines will happily heartbeat each other every 60 seconds. The heartbeats or admin messages don’t make it into BizTalk only application messages like Order’s, ExecutionReports etc will be passed on. 

To send orders the test harness has an option to construct a FIX NewOrderSingle:

image

image

At the moment as we haven’t configured the SELLSIDE if we sent an order we’ll just get back the acknowledgement.

image

To recap, using the test harness the BUYSIDEFIRM sent an Order to our message broker (BizTalk). Leveraging CBR BizTalk routed the Order back to the BUYSIDE firm and we mapped the Order to an ExecutionReport on the way out to simulate an acknowledgement. Obliviously you wouldn’t ACK every order you received but this is just a simple demo. In a future post we’ll route the Order to an Orchestration and use the Rules Engine to validate and determine where to forward it.

SELLSIDEFIRM

This entity represents a sell side firm or broker that we will be forwarding orders to from the BUYSIDEFIRM. The sell side will either reject or fill the orders which means we’ll get back more execution reports. 

FIX Details we’ll use for the receive location (from their perspective):

Property Value Description
IPAddress 127.0.0.1 This will all run on a single host
Port 9002  
SenderCompId SELLSIDEFIRM Counterparties identifier
TargetCompId MYMSGBROKER this is our BizTalk group
Initiator True They will initiate this session
Version FIX.4.2 The version of FIX that the sellside will support
HeartbeatInterval 60 seconds The interval that heartbeats will be sent, it’s important these match otherwise the sessions will logout.

 

The receive location will be pretty similar to the buyside except obviously the compids and ports will be different. For this session the SELLSIDEFIRM will initiate so BizTalk will be listening on port 9010 (configured in the adapter handler settings) and connect back to them over port 9002 as above.

image

Property Value Description
CounterParty: CompID SELLSIDEFIRM Our counterparties CompID (TargetCompId to BizTalk SenderCompID from their perspective)
IPAddress 127.0.0.1  
Port 9002  
FIXVersion FIX.4.2  
Initiator False SELLSIDE will initiate the session
HeartbeatInterval 60  
Party: CompID MYMSGBROKER  

 

For the routing the sell side send port will be configured as:

image

And filtering on the BUYSIDEFIRM’s receive port to pickup the orders:

image

Again we’ll use the RapidAddition test harness to simulate the sell side.

image

To fill the orders we can configure the sell side test harness to auto fill any orders received. This just means it will construct 4 execution reports and send them back whenever it gets an order.

image

End to End

So now if the BUYSIDE sends an order Biztalk will route it back to the BUYSIDEFIRM and map it to the execution report to simulate an ack. At the same time it will forward the Order to the SELLSIDE FIX session. The SELLSIDE test harness will auto fill the order with 4 Execution reports and send them back to BizTalk. Finally BizTalk will forward these fills back to the BUYSIDEFIRM. Sounds like a lot if going on but it’s actually pretty straight forward but if you’re still confused then check out the video.

1. BUYSIDE Sends Order

image

2. BUYSIDE Receives ACK

image

3. SELLSIDE Receives Order from BUYSIDE via BizTalk

image

4. SELLSIDE fills Order, BUYSIDE receives fills (ExecutionReports) from SELLSIDE via BizTalk

image

Hopefully this gives you an idea of how to use the BizTalk FIX adapter, it’s very easy to setup and get going with and the test harness makes testing very easy. It’s also pretty fast which hopefully shows on the video.

Friday
Apr012011

Installing and Configuring the FIX Adapter for BizTalk

This is the third post in a short series on the FIX adapter for BizTalk.

  1. FIX for BizTalk (The basics)
  2. ShortCut Architecture and Overview
  3. Installing and Configuring RapidAddition’s ShortCut
  4. Basic Routing Application using ShortCut
  5. Mapping FIX Messages
  6. More complicated scenarios and patterns using a FIX-enabled BizTalk

This post will go through installing and configuring the RapidAddition’s FIX adapter for BizTalk. The screenshots are from BizTalk Server 2009 however it’s the same process for 2010 or even 2006.

If you’re interested in getting the adapter the best place to start would be to reach out to the guys at RapidAddition.

Installing

Installing the Adapter is pretty easy. You’ll get an MSI that will install the DLL’s on to the BizTalk server. Shortcut does not support host clustering although you can install the FIX host on multiple machines so that you could quickly bring the session online if you lost a server. Each FIX session can only run in a single host and you can have multiple sessions per host.

Once you run through the installation you’ll need to configure it. There’s a configuration app in the program directory created during the installation.

image

image

The config app has 4 functions to run through. The first will install the FIX Schema and Pipeline assembles on all servers in the BizTalk group and BTSDeploy’s them. The second option will install the adapter into the registry and the third will create the RapidAddition database. The last option will make some performance tweaks to set BizTalk up for low latency.

Configuring

Adapter

The adapter is now installed. The next step is to add the adapter to the group. The config setup above really only adds the necessary registry settings for adapters like the install folder.

image

From the BizTalk Administrator, Platform Settings, Adapters (new) as shown in the screenshot above. You should see FIX in the list of uninstalled adapters. You can call it whatever you want by FIX makes a lot of sense.

Handlers

The Adapter is now installed. Once you add the adapter the default handlers will show. The send handler doesn’t have any properties.

image

As explained in the first post the send port for the FIX adapter doesn’t do much, the receive handler on the other hand does have some settings.

image

Property Description
Database The name of the RapidAddition database as installed by the config application. Default value is RapidAddition
IP Address IP Address that the adapter will listen on for inbound connections.
MaxOutgoingBatchSize Batch size for outgoing (sends) messages
Password Password for SQL Database if not using windows authentication (will be the host instance service account)
Port The port that the adapter will listen on for inbound connections.
Server The SQL server hosting the RapidAddition database
Username The username for the SQL database if not using windows authentication


 

Receive Locations

To configure the actual FIX engine most of those fields can be found in the receive  locations.

image

Property Description
BatchInterval The batch interval in milliseconds
ConnectRetryInterval The reconnect interval in milliseconds
CounterParty: CounterParty FIX details
                 CertificateID Certificate for encryption (FIX version dependent) 
                 CompID CompID of the counterparty
                 IPAddress IPAddress of the counteryparty
                 Port Port used by the counterparty
DataDictonaryVersion Data Dictionary to use for the FIX version, typically left blank unless using a custom data dictionary
FIXVersion The FIX version of the session
HeartbeatInterval Interval that heartbeats will be sent if no application messages are received or sent
IncludeHeartbeatMessages Setting to allow heartbeat messages to flow back into BizTalk
IncludeSessionRejectMessages Setting to allow session reject messages to flow back into BizTalk
IncludeTestRequestMessages Setting to allow test request messages to flow back into BizTalk
Initiator Determines how initiates the session, true we will false means the counterparty will initiate
MaxIncomingBatchSize Max number of messages in an incoming batch
MaxOutgoingBatchSize Max number of messages in an outgoing batch
Party: Party FIX details (us)
         CertificateID Certificate for encryption (FIX version dependent)
         CompID Our CompID as known to the counterparty
ResetSeqNumOnLogon Reset the sequence number upon logon, usually not set and the sequence is reset by an End of Day job
SequenceMessages Honor the sequence as they’re passed into BizTalk
SequenceMessagesTimeout Timeout of sequence
ServiceDays Days of the week that the session will start, format as MTWTFSS or MTWTFXX will mean it doesn’t logon on weekends
ServiceEndTime Time of day that the session will finish
ServiceStartTime Time of day that the session will start
StoreAndForward On by default, this tells shortcut to accept and ack outgoing messages back to biztalk regardless of there being an active session or not. This means BizTalk thinks the message went out even if the session is down. Shortcut will deal with the message once the session is up.


 

Send Ports

Send ports are a lot simpler.

image

Property Description
CounterParty: CounterParty FIX details
                 CompID CompID of the counterparty
FIXVersion The FIX version of the session
Party: Party FIX details (us)
         CompID Our CompID as known to the counterparty


These settings help Shortcut uniquely identify the session in the RapidAddition database so the application messages get sent out over the correct FIX session.

Pipeline Components

The Shortcut Pipelines need to be setup with the RapidAddition database connection string and data dictionary used to convert the messages to and from FIX and FIXML.

image

Property Description
DataDictionaryVersion the data dictionary to use in message conversion
Database Name of the RapidAddition database as setup above
Password Password for the SQL connection if not using windows authentication
Server the SQL server
UserName SQL username for the SQL connection if not using windows authentication

 

The next post will go through using shortcut for some basic routing scenarios which will demonstrate how these properties are used. 

Friday
Apr012011

RapidAddition’s ShortCut Architecture and Overview

This is part of a series on FIX-enabling BizTalk server.

  1. FIX for BizTalk (The basics)
  2. ShortCut Architecture and Overview
  3. Installing and Configuring ShortCut
  4. Basic Routing Application using ShortCut
  5. Mapping FIX Messages
  6. More complicated scenarios and patterns using a FIX-enabled BizTalk

Overview

First I want to give an overview of the ShortCut FIX Adapter, the components that get installed and give a bit more detail on what’s happening.

The diagram below gives an overview of the ShortCut adapter.

Shortcut Overview

BizTalk only ever sees the FIXML version of the messages. The conversion to and from FIX takes place in the pipelines that come with ShortCut.

The receive location not only receives FIX messages but it also sends them out. The send port just persists the converted messages into the RapidAddition database and the engine reads them and sends it over the wire. FIX is always two way communication, even if there is only one party sending application messages. Most if not all admin messages that make up the session layer are acknowledged and there are heartbeats back and forth.

A BizTalk send port only activates when a message is picked published for it. It wouldn’t be possible to have the send port be responsible for the outbound admin messages without BizTalk getting involved. Receive Ports on the other hand are always executing or polling. This then becomes the logical place to manage the session layer so it must also send out the application messages as they go over the same tcp/ip connection. Architecting the adapter this way enables the FIX engine to deal with all the FIX session layer and leaves BizTalk to only worry about the application messages.

From an Orchestration or send ports perspective it’s business as usual. They can route and send messages the same as using any other adapter. Due to some requirements of the FIX protocol the adapter has to be architected this way to save the BizTalk developer having to deal with the session layer, which is a good thing. A normal FIX session might have 6 or so different admin messages zipping back and forth, if you’re only interested in sending Orders then that’s all your application has to worry about.

Something to keep in mind though is that you’ll have to make sure the send and receive ports are running in the same host. The diagram above is actually a incorrect, but it keeps it simple for the initial explanation.

Another important FIX concept to know is that one of the parties is designated as the initiator of the session. That simply means one party always initiates the connection based on a predetermined schedule and the other is only ever listening. You configure this in the receive location which will be covered in another post.

Components

Database

As you’ve probably figured out ShortCut needs it own database. This is installed for you through a wizard which we’ll cover later. Most of the time this will be put on the SQL server that’s hosting the BizTalk databases.  If your processing a large number of messages then it might be worth putting this on a separate server. You don’t want BizTalk and ShortCut to fight for resources. Keep in mind that I’ve managed systems that process over 800,000 messages a day and it wasn’t an issue so much as an area of potential improvement if the budget was available.

For DR purposes I would consider adding the RapidAddition database to the BizTalk log shipping jobs. During a failover you’ll want the transactional state of this database to go with the whole BizTalk system. When FIX engines reconnect during the day they resync themselves based on the last processed sequence number which is stored in the database. Reconnecting from the DR site with a sequence number of 1 will cause all of the days messages to be resent which could be a nightmare.

Adapter

On the actual adapter side it’s all just a bunch of assemblies nothing all that different from what you would expect. Some are BizTalk assemblies (schemas, promoted property schemas) that you’ll reference in your projects and the rest are standard .NET dll’s.

They also give you a handy FIX test harness for testing out your application without connecting to counterparties. It can do things like auto fill orders, reject all messages and even replay FIX logs in case you have some production logs you want to test with.

Thursday
Mar312011

FIX for BizTalk (The basics)

(PDF of series)

FIX is a messaging standard and protocol that is used by the finance industry for real-time electronic exchange of securities transmissions. It was developed by the industry to facilitate the automated trading of financial instruments and is used around the world by banks, brokers, exchanges and order management and trading systems. The site fixprotocol.org has some good information to get you started.

To the uninitiated a FIX message will look a little strange but it won’t take you too long to get your head around it. Below is an example of what an Order (NewOrderSingle) might look like:

1: 8=FIX.4.2 9=118 35=D 34=23 49=OMS 56=NSDQ 52=20100721-03:16:21.679 40=1

55=MSFT 54=1 60=1999-05-31T13:20:00.000-05:00 38=100 21=1 10=169


The numbers are referred to as Tags. Tag 8 at the start is the version which in this case is FIX 4.2. Tag 35 denotes the message type, D is NewOrderSingle.  This is a BUY order (54=1) for 100 shares (38=100) of Microsoft (55=MSFT) at a market price (40=1). Tag 49 is SenderCompID and is a string that identifies the Sender of the message to the counter party or recipient. 56 is TargetCompID which is used by the counterparties FIX engine to verify the message. These values are important when setting up BizTalk which I’ll cover in a later article. 

To learn more about these tags and the messages that FIX supports there’s a handy app online: fixmate.

FIXML

The FIX protocol guys have also come up with XML versions of FIX messages which is called FIXML. The NewOrderSingle above in FIXML would look like this:

   1: <Order OrdTyp='1' Sym='MSFT' Side='1' TxnTm='1999-05-31T13:20:00.000-05:00' 
   2:         Qty='100' ClOrdID='' HandlInst='1'>
   3:    <Hdr SndgTm='1999-05-31T13:20:00.000-05:00' SeqNum='23' SID='OMS' TID='NSDQ' />
   4: </Order>


Nothing too scary, the emphasis is on small efficient messages as speed is the name of the game.

FIX Engines

A FIX engine is the application or service that enables the transmission of these messages between parties. The FIX engines connect over using a standard TCP/IP connection, they manage the session level communications (authentication, validation, heartbeats, session recovery) and the application layer (orders, executions etc).

FPL define a FIX engine as:

“A FIX engine is a piece of software that manages a network connection, creates and parses outgoing and incoming messages, respectively, and recovers if something goes wrong. A FIX engine manages the session and application layers and is the single piece of software you need in order to FIX-enable trading or order management systems. In the context of a trading system your FIX engine is the interface to the outside world, which, together with a network, connects you to outside world and allows you to trade and exchange related information in a standard fashion. Thus, to FIX-enable an application refers to the integration of a FIX engine and connection to a routing network.”

BizTalk

The folks at RapidAddition have been building FIX engines since FIX was running around in diapers. They’ve put together a very nice FIX BizTalk Adapter that FIX-enables your BizTalk Server. Your first thought is probably “I thought BizTalk was slow and FIX was about speed”, which just isn’t true. I have worked on FIX-enabled BizTalk systems manage hundreds of executions a second. Whilst there are limitations it all comes down to how you configure BizTalk and design and build the application. A lot of that is contingent on what your application has to do. For example just routing orders will be super quick but if you need to crack open the message and perform validations, transformations etc then it will come at some cost.

RapidAddition’s product is called ShortCut and is super easy to get started with. ShortCut will basically turn your Receive Locations into FIX engines. ShortCut will manage all the FIX session stuff and you just need to worry about routing or orchestrating application messages through your FIX hub. ShortCut comes with all the FIXML schemas in DLL’s for you to reference in your projects. Building applications is pretty easy and you don’t need to be a FIX weapon to do this as I hope to show in posts I’ll put together over the next few weeks.

Articles (as they’re posted)

  1. ShortCut Architecture and Overview
  2. Installing and Configuring ShortCut
  3. Basic Routing Application using ShortCut
  4. Mapping FIX Messages
  5. More complicated scenarios and patterns using a FIX-enabled BizTalk