Omit Needless Words

What William Strunk, Jr. said about writing applies to efficient messaging in connected systems. The key to high-performance communications is to omit needless messages and unnecessary fields and making every part of a message tell. To do that requires protocol expressiveness so that new structure can be created to match new workflows while leaving out what is not relevant to that workflow. To make every word tell, it is necessary that the protocol conveys the type of fields precisely, with no reformatting and with tuned value ranges. This benefits applications both at their edges and internally, allowing them to be optimized both in their message processing and in their storage of data.

Make Every Word Tell

The case for Blink is that existing protocols are limiting, in that they are either not flexible and extensible, or are not in themselves rendering the message in a concise encoding.

High-performance trading solutions are often limited by the lack of efficient standardized messaging. Products are integrated in environments where a proprietary solution that lacks adoption have little merit. We therefore see a need for improved messaging technology that is standardized and widely adopted.

Many existing protocols and frameworks meet some but not all of the needs to be processing efficient, compact, flexible, extensible, properly layered, and easy to implement. Other alternatives are just widely adopted and continue to be used because of the switching cost and for lack of good alternatives.

We designed Blink to do all of the above with as few compromises as possible. We are releasing Blink to promote adoption of an open alternative.

Blink can be applied in several layers in the communications stack, such as the application and the session layers.

FAST

In 2005, FIX Protocol Ltd (FPL) formed the Market Data Optimization Working Group (MDOwg) with the goal of defining a new message protocol for efficient representation of financial market data. Pantor joined the MDOwg to contribute in the development process. Rolf Andersson came up with the original design for what later became the FAST protocol standard version 1. David Rosenborg continued the standardization work by authoring the FAST version 1.1 specification.

FAST was able to reduce the cost of distributing financial market data by drastically decreasing the network bandwidth requirements. It was also considered for order routing, but was never adopted in this usage scenario. FAST has also been viewed as complex and difficult to implement efficiently.

Over time, it has become obvious that FAST, as originally defined, solves a fairly narrow problem and would need to be complemented by another protocol or be enhanced to simplify its use and to handle a wider range of usage scenarios.

Simplified FAST

In 2010, Anders Furuhed together with David and Rolf started exchanging ideas on a simplified subset of FAST.

The original work on simplified FAST was based on three ideas:

• Focus on workflow-driven message structures
• Tradeoff between compactness and other properties
• Use of an integrated message schema

Focus on a workflow-driven message structure

The key to creating an efficient application protocol layer is to capture the business requirements in an expressive, to the point, workflow-driven message structure, using a type system to restrict value ranges and expressing the intent of the source in a way that eliminates redundant information.

As a contrast, FAST has been used to optimize the messages of existing flows by specifying templates that identify the same intent, post fact, effectively creating expressiveness that the protocol being rendered onto FAST lacked in the first place. This meant that the intent known to the producer of messages was thrown away and recaptured by creating templates, instead of redesigning the original protocol in a way that would have eliminated the redundancy at the source.

Tradeoff between compactness and other properties

The compactness of FAST could be traded for other features in the setting of order flows, where messages are not clustered in the same way as in the market data setting. We later realized that the same can apply to market data if it is conveyed in a more concise way. To give an example, by letting a matching engine signal the notion of a matching event instead of expressing a sequence of unrelated resulting sub events ("partial fills"), the resulting messages become more concise and therefore more compact.

Use of an integrated schema

A protocol that integrates the definition of message structure in a schema has an advantage over a protocol like FAST that expresses template variations of messages defined elsewhere. The first advantage follows from making the protocol comprehensive and self sufficient. That makes it easier to write and understand a complete specification of the protocol. The second advantage is that new structure can be defined in a fully integrated way. This allows all aspects of meta data to be part of the stream but separately from messages. The protocol becomes a meta protocol, a protocol for defining protocols, without compromising encoding efficiency.

Early work

An early prototype implementation presented at the FPL Nordic Briefing in November 2010 showed that the protocol is processing efficient and compact.

Recent work

In September 2012, David Rosenborg ironed out the details of the protocol with the help of Rolf and Anders and wrote a first complete implementation of what became called Blink. The first draft of the Blink specification was completed early November 2012. The first open source stack implementing Blink was released in May 2013.