High Performance Trading/Algo Speed with Wombat Design and

62 Pages·2008·1.5 MB·English

High Performance Trading/Algo Speed
with Wombat Design and Implementation
Guide
January 24, 2008
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: [phone]
800 553-NETS (6387)
Fax: [phone]
Text Part Number: OL-15617-01
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE
PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE
WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING,
USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR
INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY
TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE
DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD
CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED
BY CISCO.
CCVP, the Cisco Logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of
Cisco Systems, Inc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo,
Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step,
Follow Me Browsing, FormShare, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, iPhone, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick
Study, LightStream, Linksys, MeetingPlace, MGX, Networking Academy, Network Registrar, Packet, PIX, ProConnect, RateMUX, ScriptShare, SlideCast, SMARTnet,
StackWise, The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and
certain other countries.
All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship
between Cisco and any other company. (0612R)
High Performance Trading/Algo Speed with Wombat Design and Implementation Guide
© 2007 Cisco Systems, Inc. All rights reserved.
C O N T E N T S
Introduction 1-1
Target Audience 1-3
Target Market 1-4
Automated Trading Benefits 1-4
Automated Trading Architecture 1-6
Concept Features 1-10
Tested Components 1-11
Servers 1-11
Networking 1-12
Operating System 1-12
Test Implementation Framework 1-13
Testing Topology 1-13
Testing 1-14
Methodology 1-14
Test Setup 1-14
Procedures 1-15
Data Observations 1-16
Time Synchronization 1-16
Limitations 1-16
Testing Results 1-17
Mean Latency 1-17
Latency Dispersion 1-22
Appendix A—Device Configuration 1-28
Catalyst Switch Configuration 1-28
SFS 7000 Configuration (Core) 1-40
Appendix B—Building and Configuring Switches 1-42
Definitions 1-42
The Basics 1-42
Installation Task and Timing Overview 1-43
The Very First Thing That You Do: Plan 1-43
Install Interface Cards in the Hosts 1-48
Rack and Cable All Hardware 1-48
Write Down Your Cabling Connections 1-48
High Performance Trading/Algo Speed with Wombat Design and Implementation Guide
OL-15617-01 i
Contents
Configure Ethernet AtTributes of Leaf Switches 1-49
Configure Ethernet AtTributes of Core Switches 1-49
Validate the Ethernet Management NEtwork 1-50
Set Up SE Tools on a Ethernet-attached Host 1-50
Perform a Switch Chassis Inspection 1-50
Perform a Physical Inspection 1-50
(Optional) Record Leaf Switches and Hosts 1-50
Disable Uplinks on Leaf Switches 1-51
Install Host-Side Drivers and Configure IP Addresses to InfinIBand Ports on Hosts 1-51
Troubleshoot “Bring Up” Pod 1-53
Run Step Troubleshoot “Bring Up” Pod On All Pods 1-55
Connect “bring up” Pod to Core Switches One at a Time 1-55
Connect Pods to Core Switches 1-55
Troubleshooting after Pruning 1-55
High Performance Trading/Algo Speed with Wombat Design and Implementation Guide
ii OL-15617-01
High Performance Trading/Algo Speed with
Wombat Design and Implementation Guide
Introduction
Automated trading and new regulatory demands are the two main forces behind the changes in Financial
Markets. Firms are trying to maintain their competitive edge by constantly changing their trading
strategies and increasing the speed of trading. New financial products, business models, and trading tools
require a super fast response.
Automated trading is creating a faster trading cycle, in which milliseconds matter. The faster your
trading infrastructure is the better chance you have of hitting your price points in a very dynamic market.
This shift from manual to automated trading is creating a huge strain on the information infrastructure
of financial firms, because it requires both speed and handling huge volumes of data, with maximum
reliability and predictability. The front-office systems of trading departments are the most latency
sensitive and the most visible in the value chain. This is where the automated trading happens and where
Cisco is offering the High Performance Trading (HPT) Algo Speed solution.
HPT Algo Speed solution enables automated trading applications to communicate fast and handle high
volumes of market data in a predictable and reliable manner. For example, to gain a competitive edge in
the marketplace, portfolio and risk managers must be able to access real-time financial information and
use technical indicators to buy and sell equities and exotic investments. As new investment products are
introduced, the need to obtain an accurate view of a fund’s value and its related risk is significantly
increased. These growing business necessities are creating enormous stress on the current compute
infrastructures, where highly computational algorithms in mission-critical applications are unable to
scale effectively.
Cisco is uniquely positioned to address this problem with its high-performance computing solutions,
which include both Ethernet and InfiniBand technologies. Cisco’s broad solution portfolio delivers high
speeds, low latencies, open standards, and high system availability and allows financial customers to
deploy the right infrastructure for their application.
To solve many of the emerging application requirements, Cisco offers InfiniBand and high-density 10
Gigabit Ethernet solutions, which are optimized for the most-demanding financial applications. The
InfiniBand 4X DDR (double data rate) technology in the Server Fabric Switching (SFS) family can
provide throughput rates of up to 20 Gbps. This ultra-low-latency computing fabric provides native
remote direct memory access (RDMA) capabilities, to share computational power across multiple CPUs
and ensure maximum cluster performance. RDMA has the additional benefit of allowing inter- CPU and
Corporate Headquarters:
Cisco Systems, Inc., 170 West Tasman Drive, San Jose, CA 95134-1706 USA
Copyright © 2007 Cisco Systems, Inc. All rights reserved.
Introduction
memory read/writes, as well as kernel bypass. Applications that specifically support Message Passing
Interface (MPI) or Open Fabrics messaging transports can effectively achieve latencies of less than 10
microseconds.
The InfiniBand fabric can also be seamlessly integrated with existing Ethernet networks by using SFS
3000 Series switches-this eliminates any interoperability concerns. The Cisco Catalyst® 6500 Series
switches offer a high-density 10 Gigabit Ethernet solution with low latency that is ideal for large
Ethernet clusters. The flexibility of multiple-grid computing designs that can adapt to any grid
environment allows an organization to scale quickly with growing business demands. Financial market
data providers and consumers are aggressively preparing for upcoming changes in the industry driven
by the Regulation National Market System (Reg NMS), the Markets in Financial Instruments Directive
(MiFID), and FIX Adapted for Streaming (FAST). A utility computing model is necessary to help
organizations respond to these changing market conditions and skyrocketing data rates. A utility
computing model constructed with Cisco InfiniBand fabric and the VFrame management platform can
help manage virtualized computing and network resources. Capacity can be added on demand or as
required by business policies. As clusters become grids, Cisco can quickly make use of servers that allow
an organization to scale up its computing resources faster and more cost-effectively. New services and
extra compute power can be added automatically, on the fly, to maximize the utilization of servers that
typically operate below optimum capacity. This intelligent HPC network fabric helps reduces the total
cost of ownership because resources are used more efficiently. Further, it provides the architecture for
consolidating and virtualizing resources, enabling the evolution to an automated system that is able to
dynamically respond to changing business needs.
High Performance Trading/Algo Speed with Wombat Design and Implementation Guide
2 OL-15617-01
Introduction
Figure 1 shows the general automated trading solution environment.
Figure 1 General Automated Trading Solution Environment
Nasdaq
NYSE CME
INET OPRA
IB Fabric supports multicast in
HW; SP and DAL provide RDMA
performance for applications
Direct Exchange
using TCP and UDP
Primary Circuit Feeds
Multicast Router/
Firewall/Screening TCP or UDP;
Unicast/Multicast
Feed Handlers for
each Exchange
(Reuters, Bloomberg,
Wombat)
UDP Multicast or TCP
(TIBCO-RV, RMDS, 29West)
Intermediary
Processing Engines
TCP or UDP
Unicast/Multicast
(TIBCO-RV, RMDS, 29West)
Analytics/
Execution Engines
Transaction
Processing System
1
5
2
3
2
2
Target Audience
This document is intended to be used by Cisco Systems Engineers, Advanced Services Engineers,
partners, and clients who work on trading floor infrastructure projects. The document was created based
on the testing performed by STAC research to quantify the performance gains of low latency computing
in an HPT environment. The test environment was limited and not necessarily characteristic of a
production environment, but the concepts are the same. The information contained in this document can
be used to identify, build and test a production prototype environment.
Note This environment was tuned specifically for the testing performed. Note that any change from what was
specifically tested here can affect your results. It is important to keep this in mind as you build your test
environment and go through the associated test plans.
High Performance Trading/Algo Speed with Wombat Design and Implementation Guide
OL-15617-01 3
Introduction
Target Market
Financial services firms require vast amounts of computing power to run their business. They use
homegrown applications or increasingly ISV applications to do these computations. However, these
applications were originally built for SMP machines or hard-coded clusters of a certain size. The net
result is that computations takes hours and days, while the business needs it to happen in seconds and
minutes.
Further, the average utilization of the server farm is very low ( less than 10 percent). Given a fixed IT
budget, this means the firm does a lot less computation than it would like to. Fewer scenarios are run and
modeling is done to less accuracy. Some businesses do not use modeling at all due to lack of resources
Solving these problems can provide a competitive advantage to the customer. Specifically, the following
businesses are the ones most affected:
• Front office—Pricing and hedging of derivatives, foreign exchange options, and other structured
financial products
(cid:127) Mid office—Risk analysis of portfolio, counter-party credit risk, enterprise risk, proprietary desk
analytics
(cid:127) Back office—Fraud detection, global treasury
Automated Trading Benefits
Automated trading benefits include the following:
(cid:127) Delivers low-latency and high speed Ethernet and InfiniBand interconnects for financial
applications such as trading floors and market data feed
Automated trading makes reducing latency and increasing performance essential in a market data
environment. Feed handlers receive real-time market data feeds from sources such as Options Price
Reporting Authority (OPRA), NASDAQ, and electronic communications networks (ECNs), and
these feeds need to be “normalized” before being distributed to users. Data normalization allows
feeds to be standardized through dedicated feed processors and entered into a uniform database
model. The uniform access to multiple normalized market data feeds facilitates data distribution to
end-users and ensures data consistency throughout the organization. The high server-to-server
traffic that occurs as these computations are performed means that a cluster of servers with the
lowest possible latency interconnect is needed to reduce delay in delivering the market data.
Through the use of RDMA technology, an application can offload all communications management
to the InfiniBand host channel adapter, which allows more CPU cycles to be spent on processing,
rather than communications. Cisco’s innovative HPT solution creates a high-performance server I/O
fabric, achieving ultra-low-latency performance to support the growing computing needs of market
data feed handlers and other trading floor applications.
(cid:127) Minimizes latency in each component of delivery platform
It is imperative that latency be minimized when delivering time-sensitive data. As the data traverses
the different components of a trading platform—including market data delivery, order routing, and
execution—an HPC environment addresses the speed-sensitivity requirements by providing a lowest
latency interconnect, so that raw computational power can be used in clusters to deliver the fastest
response possible.
(cid:127) Helps prepare for new regulations that drives high market-data volumes
High Performance Trading/Algo Speed with Wombat Design and Implementation Guide
4 OL-15617-01
Introduction
Regulatory changes such as Reg NMS generate more quote, order, and cancel/replace messages as
equity firms adapt to more electronic business processes. The subpenny pricing rule also increases
demands on the supporting infrastructure. MiFID, which goes into effect in Europe next year, is
expected to lead to higher data volumes as well, since investment banks that internalize trades will
be required to publish their pre-trade quotes electronically. An InfiniBand HPC environment
provides a secure, scalable solution to meet the growing needs of the financial services industry.
(cid:127) Increases competitive edge by incorporating FAST protocol for lowest-latency connections
As financial industry experts predict relentless growth in market-data volumes, organizations are
preparing for implementation of the FAST Protocol. FAST offers a lower-latency feed that uses a
data compression technology. Exchanges are planning to use FAST to deliver new products in areas
such as derivatives and equities. However, to support the new, rapid market data message rates and
deliver the quickest-possible trade execution, a grid computing model is needed to parse the data
from the feeds and then deliver it to consumers. The InfiniBand HPC solution can provide the lowest
latency transport while providing the bandwidth to sustain increased market volumes.
(cid:127) Provides flexibility to support a service-oriented architecture based on industry-standard protocols
The Cisco HPC solution adheres to industry standard protocols such as Open MPI and Open Fabrics.
This allows customers to leverage the true high-performance, ultra low-latency characteristics of the
InfiniBand fabric. An application environment that supports these industry protocols benefits from
the open architecture, as it becomes part of a service-oriented architecture strategy for the adaptive
enterprise. This shortens the time to market for new financial products, providing a competitive
edge.
(cid:127) Allows the use of common tools to manage Ethernet and InfiniBand networks
The VFrame management platform enables the delivery of utility computing into the data center
environment. This increases the ability to rapidly provision shared-server and I/O resources on
demand. By managing and orchestrating diverse Ethernet and InfiniBand resources, a financial
organization can become more agile-adapting easily to rapidly changing market conditions. In
addition, just intime provisioning reduces operational costs by automating regular tasks. Since
InfiniBand creates a high-speed fabric that is shared by all the nodes participating in it, downtime
can be quickly averted by reallocating resources to different resource pools.
(cid:127) Supports the increasing trend toward a utility computing model to support heavy computations
As market-data levels continue to rise and financial organizations look to expand their product
portfolios, increasing raw computational power is required to support the algorithms needed for
portfolio performance analytics, Monte Carlo simulations, value calculations, and risk profiling of
trades. Grid services virtualize computing silos that underperform or are underutilized and makes
them well-balanced, fully optimized enterprise backbones. The Server Fabric HPC infrastructure
can be scaled quickly, because additional computing power can be added to the grid dynamically to
support business processes with increasing demands. The combination of Server Fabric Switching
and VFrame allows for effective consolidation, virtualization, and automation of resources that
deliver instantaneous return on investment.
High Performance Trading/Algo Speed with Wombat Design and Implementation Guide
OL-15617-01 5
Automated Trading Architecture
Automated Trading Architecture
Figure 2 shows the general automated trading network architecture.
Figure 2 General Automated Trading Network Architecture
FC Web
NAS
Storage Clients
LAN/Routed
Intranet
SAN
CCCaaatttaaalllyyysssttt SSSwwwiiitttccchhh
MMMDDDSSS SSSwwwiiitttccchhh
FFFCCC
Cisco
IO SFS 3012
IB
Cisco
Core
SFS 7008/12/24
IB
Cisco
Leaf
SFS 7000
IB
Servers
GE 12 Hosts
Management 52
2
3
2
2
High Performance Trading/Algo Speed with Wombat Design and Implementation Guide
6 OL-15617-01