sop - master
TRANSCRIPT
SOP - Master Page 1
Standard Operating Procedures
Version 02
SOP Sample for LinkedIn Display Purposes. No content included.
Author: Dan Schwartz Approved: 03-September-2013
Approved By: Managing Director: Mike Kreutzjans
Managing Director: Dan May
Director of Operations: Dan Schwartz
SOP - Master Page 2
REVISIONS
The following revisions were applied to the original version:
Revision Date Version No. Author Description
02-Sep-2008 1.00 Dan Schwartz Created initial template
07-Oct-2008 1.01 Dan Schwartz Updated template & creation of first version of the table of contents.
08-Oct-2008 1.02 Dan Schwartz Incorporated additional content sent from Ron Lindstrom into the outline.
03-Dec-2008 1.03 Dan Schwartz Updated the PDU and IDC Developer Notices sections.
26-Dec-2008 1.04 Dan Schwartz
Completed initial versions of sections: 1 – Overview 2 – Scope 3 – About SpryWare 4 – Service Level Agreement (SLA)
29-Dec-2008 1.05 Dan Schwartz
Section & Change: 4 – SLA Updated to match current SLA 7 – Data & Hosting – Updated 8 – Customer Service Guide – Updated 9 – Release Management – Passed ownership of this section to Ron Lindstrom.
13-Jan-2009 1.06 Dan Schwartz Reworked the general outline. Too many updates to document. Doc now ready for distribution to Ron, Terry & Tony. Specific procedures can now be added.
13-Jan-2009 1.07 Dan Schwartz Updates following a high level review with Ron. Document distributed to Ron, Terry, Tony, Dan M, Mike K and SharePoint.
21-Jan-2009 1.08 Dan Schwartz Wrote Cisco Systems VPN section.
Client Side MIS.XML
PDU Outlet Configuration
12-May-2009 1.09 Dan Schwartz
Server Side MIS.XML
Remote Access o Assent o Base-2 o BNY o Flow Trading o Hold Brothers o OptionsXpress o Sterling o UNX
MISVerify
25-Oct-2009 1.10 Dan Schwartz
MIS Upgrade Procedures New Employee Procedures Added CRM Added Infrastructure Section
SOP - Master Page 3
02-Aug-2010 1.11 Dan Schwartz
Reconfigured & updated Email/SharePoint/CRM under Intermedia
Removed content covered in stand alone docs
Added Bugzilla to the Toolsets section.
12-Jan-2011 1.12 Dan Schwartz Server Decommission Procedures
Iron Mountain
03-Sep-2013 02 Dan S. Updated Org Chart
SOP - Master Page 4
TABLE OF CONTENTS
1. Overview ............................................................................................................................................. 10
2. Supplementary Documentation .......................................................................................................... 11
2.1 Stand Alone Document Directory ............................................................................................... 11
2.1.1 SpryWare Technical Overview ............................................................................................ 11
2.1.2 SOP – Active Directory ........................................................................................................ 11
2.1.3 SOP – Backup and Restore Procedures ............................................................................... 11
2.1.4 SOP – BGInfo ....................................................................................................................... 11
2.1.5 SOP – Bugzilla ...................................................................................................................... 11
2.1.6 SOP – Client Remote Access Procedures ............................................................................ 11
2.1.7 SOP – CRM User Manual ..................................................................................................... 11
2.1.8 SOP – Data Editing............................................................................................................... 11
2.1.9 SOP – DMS Prep .................................................................................................................. 11
2.1.10 SOP – DMS Linux MIS Build Procedures .............................................................................. 11
2.1.11 SOP – DMS Windows MIS Build Procedures ....................................................................... 11
2.1.12 SOP – Entitlement Procedures ............................................................................................ 11
2.1.13 SOP – ESXi QuickGuide ........................................................................................................ 11
2.1.14 SOP – General Office ........................................................................................................... 11
2.1.15 SOP – HDS ........................................................................................................................... 11
2.1.16 SOP – IDC Developer Notices .............................................................................................. 11
2.1.17 SOP – Intel to SloarFlare NIC Upgrade Procedures ............................................................. 11
2.1.18 SOP – Iron Mountain ........................................................................................................... 11
2.1.19 SOP – Nagios Manual .......................................................................................................... 11
2.1.20 SOP – New Client Procedures ............................................................................................. 11
2.1.21 SOP – New Employee Procedures ....................................................................................... 11
2.1.22 SOP – SharePoint ................................................................................................................ 11
2.1.23 SOP – SLA & Customer Service Guide ................................................................................. 11
2.1.24 SOP – VPN Guide ................................................................................................................. 11
3. Scope ................................................................................................................................................... 12
4. About SpryWare .................................................................................................................................. 13
4.1 Locations ..................................................................................................................................... 13
4.2 Telephone ................................................................................................................................... 13
4.3 Email ............................................................................................................................................ 13
4.4 Hours of Operation ..................................................................................................................... 13
4.4.1 Holiday Schedule – NYSE, Canada & London Stock Exchange ............................................ 14
4.5 Org Chart ..................................................................................................................................... 15
4.6 Products ...................................................................................................................................... 15
4.6.1 The Direct Market Server (DMS) ......................................................................................... 15
4.6.2 HDS ...................................................................................................................................... 17
5. Service Level Agreement ..................................................................................................................... 18
SOP - Master Page 5
5.1 Scope ........................................................................................................................................... 18
5.2 Notification and Credits .............................................................................................................. 18
5.3 Exclusions .................................................................................................................................... 18
5.4 Credit Types ................................................................................................................................ 18
5.5 Service Availability Metrics ......................................................................................................... 19
5.5.1 Service Point Problem Resolution Time .............................................................................. 19
5.5.2 Request for Reasons for Outage Guarantee ....................................................................... 19
5.5.3 De Facto Outages ................................................................................................................ 20
5.6 Applicability ................................................................................................................................. 20
6. Customer Service Guide ...................................................................................................................... 21
6.1 Overview ..................................................................................................................................... 21
6.2 Network Monitoring ................................................................................................................... 21
6.3 Hours of Operation ..................................................................................................................... 21
6.4 Email Support .............................................................................................................................. 22
6.5 Phone Support ............................................................................................................................ 22
6.6 Problem Severity, Communication and Response Definitions ................................................... 22
6.6.1 Definitions ........................................................................................................................... 22
7. Spryware Website ............................................................................................................................... 23
8. Toolsets ............................................................................................................................................... 24
8.1 Active Directory .......................................................................................................................... 24
8.2 Intermedia ................................................................................................................................... 25
8.2.1 Overview ............................................................................................................................. 25
8.2.2 Email – Exchange/Outlook .................................................................................................. 25
8.2.3 SharePoint ........................................................................................................................... 25
8.2.4 CRM ..................................................................................................................................... 29
8.3 SalesForce ................................................................................................................................... 29
8.3.1 Overview ............................................................................................................................. 29
8.4 CVS .............................................................................................................................................. 30
8.4.1 Overview ............................................................................................................................. 30
8.4.2 Process & Procedures ......................................................................................................... 30
8.5 Nagios .......................................................................................................................................... 30
8.6 BGINFO ........................................................................................................................................ 31
8.7 MISVerify ..................................................................................................................................... 31
8.8 VPN .............................................................................................................................................. 32
8.9 Bugzilla ........................................................................................................................................ 32
9. Direct Market Server (DMS) ................................................................................................................ 33
9.1 Overview ..................................................................................................................................... 33
9.2 Technical Requirements .............................................................................................................. 33
9.2.1 Hardware............................................................................................................................. 33
9.2.2 Operating System ................................................................................................................ 33
9.2.3 Network .............................................................................................................................. 33
SOP - Master Page 6
9.3 MIS Build Procedures .................................................................................................................. 33
9.4 MIS.XML ...................................................................................................................................... 34
9.4.1 Client Side ........................................................................................................................... 35
9.4.2 Server Side (Data Server) .................................................................................................... 36
9.4.3 Server Side (Proxy) .............................................................................................................. 37
9.5 MIS Decommission Procedures .................................................................................................. 39
9.6 Utilities ........................................................................................................................................ 40
9.6.1 Tableviewer ......................................................................................................................... 40
9.6.2 MDUMP ............................................................................................................................... 40
9.6.3 Tail ....................................................................................................................................... 40
9.6.4 MISTOP/MISTART ............................................................................................................... 40
9.6.5 Snapshot.............................................................................................................................. 40
9.7 Process & Procedures ................................................................................................................. 41
9.7.1 SpryWare Upgrade .............................................................................................................. 41
9.7.2 Data Editing ......................................................................................................................... 41
9.7.3 Operating System Patches .................................................................................................. 43
9.7.4 Conflation ............................................................................................................................ 43
9.7.5 Arbitration ........................................................................................................................... 43
9.7.6 Front End/Back End/Proxy/MIG.......................................................................................... 44
9.7.7 MIS Redundancy/Fail Over.................................................................................................. 44
9.7.8 Server Detail/BGInfo ........................................................................................................... 44
9.7.9 Backup & Restore Procedures............................................................................................. 44
9.7.10 External RAID Arrays ........................................................................................................... 44
9.7.11 Hardware Upgrades ............................................................................................................ 44
9.7.12 Hardware Failures ............................................................................................................... 44
9.8 Troubleshooting Guide ............................................................................................................... 45
9.8.1 Client Side Logging .............................................................................................................. 46
10. Infrastructure ...................................................................................................................................... 48
10.1 ADM ............................................................................................................................................ 48
10.2 ESXi .............................................................................................................................................. 48
10.3 NAS .............................................................................................................................................. 48
10.4 Dell MD1000 ............................................................................................................................... 48
11. Market Data ........................................................................................................................................ 49
11.1 Overview ..................................................................................................................................... 49
11.2 SharePoint Market Data Calendar .............................................................................................. 49
11.3 Market Data Calendar Sources ................................................................................................... 50
11.3.1 NASDAQ OMX ..................................................................................................................... 50
11.3.2 Clearnet Calendar ............................................................................................................... 51
11.3.3 Interactive Data Corporation (IDC) Developer Notices ....................................................... 51
12. SpryWare Hosted Data Centers .......................................................................................................... 52
12.1 Overview ..................................................................................................................................... 52
SOP - Master Page 7
12.2 Redundancy ................................................................................................................................ 52
12.3 Disaster Recovery ........................................................................................................................ 52
12.4 Support........................................................................................................................................ 52
12.5 Customer Care Area .................................................................................................................... 53
12.6 Site Access ................................................................................................................................... 53
12.7 Data Center Infrastructure .......................................................................................................... 53
12.7.1 Overview ............................................................................................................................. 53
12.7.2 Physical Security .................................................................................................................. 53
12.7.3 Network Operations Center (NOC) ..................................................................................... 53
12.7.4 Power Systems & Distributed Redundancy ........................................................................ 54
12.7.5 Customer Power ................................................................................................................. 54
12.7.6 Network (Internet & Carrier Detail) .................................................................................... 54
12.7.7 Environmental Controls ...................................................................................................... 54
12.7.8 Fire Protection..................................................................................................................... 55
12.7.9 Flood Control....................................................................................................................... 55
12.7.10 Earthquake Preparedness ............................................................................................... 55
12.8 SpryWare Infrastructure ............................................................................................................. 56
12.8.1 Overview ............................................................................................................................. 56
12.8.2 PDU ..................................................................................................................................... 56
12.8.3 KVM ..................................................................................................................................... 56
12.8.4 Dell DRAC ............................................................................................................................ 56
12.8.5 Network .............................................................................................................................. 56
12.8.6 Cabling ................................................................................................................................. 56
12.8.7 Monitoring .......................................................................................................................... 56
12.9 Policies & Procedures ................................................................................................................. 56
12.9.1 New Cabinet Build Out ........................................................................................................ 56
12.9.2 Cable a New MIS ................................................................................................................. 57
12.9.3 PDU Outlet Configuration ................................................................................................... 58
12.9.4 Cisco 4948 Installation & Configuration ............................................................................. 59
12.9.5 Cisco 4948 Port Configuration ............................................................................................ 59
12.9.6 Dell DRAC ............................................................................................................................ 59
12.9.7 Chicago Equinix – Specific Procedures ................................................................................ 59
12.9.8 New York SIAC – Specific Procedures ................................................................................. 62
13. Client Support ..................................................................................................................................... 65
13.1 Overview ..................................................................................................................................... 65
13.2 System Monitoring ...................................................................................................................... 65
13.3 Remote Access ............................................................................................................................ 66
14. Communication & Training ................................................................................................................. 67
14.1 Reporting..................................................................................................................................... 67
14.1.1 Build Lists ............................................................................................................................ 67
14.1.2 Release Notes ...................................................................................................................... 67
SOP - Master Page 8
14.1.3 Defect/Bug Reports ............................................................................................................. 67
14.1.4 Status Reports ..................................................................................................................... 67
14.2 Status Meetings .......................................................................................................................... 67
14.3 Training Documentation ............................................................................................................. 67
14.3.1 KnowledgeQuest ................................................................................................................. 67
15. Technical Operations .......................................................................................................................... 68
15.1 Overview ..................................................................................................................................... 68
15.2 System Monitoring ...................................................................................................................... 68
15.2.1 Server Availability & Performance ...................................................................................... 68
15.2.2 Application Software ........................................................................................................... 68
15.2.3 Website ............................................................................................................................... 68
15.3 Problem Management ................................................................................................................ 68
15.3.1 Overview ............................................................................................................................. 68
15.4 Problem Description & Resolution.............................................................................................. 69
15.4.1 ALERT !!! mail.spryware.local ques is x items (ADM1) ....................................................... 69
15.5 Change Management .................................................................................................................. 70
15.5.1 Overview ............................................................................................................................. 70
15.5.2 Customer Request ............................................................................................................... 70
15.5.3 Internal Changes ................................................................................................................. 70
15.5.4 Policy & Procedures ............................................................................................................ 70
15.6 Process Management ................................................................................................................. 71
15.6.1 Holiday Procedures ............................................................................................................. 71
15.6.2 Upgrade Candidate Identification ....................................................................................... 71
15.6.3 Client Communications ....................................................................................................... 71
15.6.4 Application of Patches ........................................................................................................ 71
15.6.5 Prine .................................................................................................................................... 71
15.6.6 ADM1-3 ............................................................................................................................... 71
15.6.7 NAS ...................................................................................................................................... 71
15.7 Inventory Control ........................................................................................................................ 71
15.8 Contingency Planning.................................................................................................................. 71
15.9 Issue Tracking .............................................................................................................................. 71
15.9.1 Defect – Functionally Correct, Inelegant Implementation ................................................. 72
15.9.2 Bug – Functionally Incorrect or Non-Functional ................................................................. 72
16. Customer Support ............................................................................................................................... 73
17. Quality Assurance ............................................................................................................................... 74
17.1 Scope Definition .......................................................................................................................... 74
17.2 OK to Develop ............................................................................................................................. 74
17.3 Unit Testing ................................................................................................................................. 74
17.4 Push to QA .................................................................................................................................. 74
17.5 Quality Control Testing ............................................................................................................... 74
17.6 Push to Beta (Controlled Production Release) ........................................................................... 74
SOP - Master Page 9
17.7 User Acceptance ......................................................................................................................... 74
17.8 Push to Production ...................................................................................................................... 74
17.9 Issue Reporting/Resolution ......................................................................................................... 74
18. Development ....................................................................................................................................... 75
19. Sales & Marketing ............................................................................................................................... 76
20. General Office ..................................................................................................................................... 77
20.1 New Employee ............................................................................................................................ 77
20.2 Telephone System ....................................................................................................................... 78
20.3 Laptop/Desktop .......................................................................................................................... 78
20.4 Printers ........................................................................................................................................ 78
20.5 GoToMeeting .............................................................................................................................. 79
21. Preferred Vendor List .......................................................................................................................... 80
21.1 Vendor Support ........................................................................................................................... 80
SOP - Master Page 10
1. OVERVIEW
The SpryWare Standard Operating Procedures (SOP) document outlines the primary support
processes required for the operation of the company’s technical infrastructure. This includes the
offices located in Chicago & New York, the two data centers as well as support for the servers
located at customer sites.
This document will provide a framework for the continued documentation of operating procedures
as service offerings grow and the environment expands.
This document is broken down into multiple sections. These sections will document:
Site Locations and Contact Information
Service Level Agreement (SLA)
Process & Procedures
Hosted Data Center Model
Toolsets available for running the business
Communication & Training
SpryWare Market Information Server Solution Detail
Preferred Vendor List
This document does not include the Employee Handbook, SDK, etc. However, these documents can
be found on the SpryWare SharePoint site.
SOP - Master Page 11
2. SUPPLEMENTARY DOCUMENTATION
This Master document begins with a Table of Contents detailing the layout of the SpryWare
Standard Operating Procedures. Each subsequent topic is then covered in the sections that follow.
In some cases, the subject matter requires a lengthy write up; too much detail for this single Master
document. In other cases, the subject matter will be referenced frequently as part of day to day
operations. To address this, separate/stand alone documents are included as part of the overall SOP
documentation effort. In each case these stand alone documents are referenced in the master
document. Each stand alone document is titled using the following format:
SOP – Subject Matter Description v##
2.1 STAND ALONE DOCUMENT DIRECTORY
2.1.1 SpryWare Technical Overview
2.1.2 SOP – Active Directory
2.1.3 SOP – Backup and Restore Procedures
2.1.4 SOP – BGInfo
2.1.5 SOP – Bugzilla
2.1.6 SOP – Client Remote Access Procedures
2.1.7 SOP – CRM User Manual
2.1.8 SOP – Data Editing
2.1.9 SOP – DMS Prep
2.1.10 SOP – DMS Linux MIS Build Procedures
2.1.11 SOP – DMS Windows MIS Build Procedures
2.1.12 SOP – Entitlement Procedures
2.1.13 SOP – ESXi QuickGuide
2.1.14 SOP – General Office
2.1.15 SOP – HDS
2.1.16 SOP – IDC Developer Notices
2.1.17 SOP – Intel to SloarFlare NIC Upgrade Procedures
2.1.18 SOP – Iron Mountain
2.1.19 SOP – Nagios Manual
2.1.20 SOP – New Client Procedures
2.1.21 SOP – New Employee Procedures
2.1.22 SOP – SharePoint
2.1.23 SOP – SLA & Customer Service Guide
2.1.24 SOP – VPN Guide
SOP - Master Page 12
3. SCOPE
This document details all aspects associated with the support of SpryWare’s technical infrastructure
ranging from the company internal functions, like the telephone system and Microsoft Exchange, all
the way to the customer facing Market Information Server (MIS) and associated networking.
The document begins with a general overview of SpryWare highlighting information that can be
found in the About Us section of the SpryWare corporate website located at:
http://www.spryware.com
The next section documents the Service Level Agreement (SLA). The SLA records a common
understanding about services, priorities, responsibilities, guarantees and warranties. Each area of
the service scope defines the level of service SpryWare is committed to achieving.
A detailed overview of the various toolsets SpryWare relies on follows. These toolsets include, but
are not limited to Exchange, SharePoint, CRM, Nagios-based Network & Event Monitoring and
SalesForce.
Also included as in-scope for this document are sections detailing Technical Infrastructure, Release
Management, Quality Assurance and Customer Support. Processes & Procedures are documented
along with a list of approved vendors.
Areas outside the scope of this document include the Employee Handbook, SDK, etc. However,
these documents can be found on the SpryWare SharePoint site located at:
http://spryware.hostpilot.com
SOP - Master Page 13
4. ABOUT SPRYWARE
As detailed in the About Us section of the company website, SpryWare provides standardized
market data via direct exchange feeds. SpryWare’s flagship product, The Market Information Server
(MIS), represents the latest and fastest in the low-latency, DMA marketplace. Working in a niche
field, the company is 100% focused on delivering market data, faster.
4.1 LOCATIONS
SpryWare operates out of two locations situated in the heart of the Chicago and New York financial
districts. The corporate headquarters are located at the Chicago location. The office addresses are
listed below:
Chicago One North LaSalle Street Suite 4300 Chicago, IL 60602
New York One Liberty Plaza 23rd Floor New York, NY 10006
4.2 TELEPHONE
Corporate HQ Chicago IL: (312) 922-7779
Chicago Fax: (312) 922-5158
New York Office: (212) 201-5515
4.3 EMAIL
Customer Support: [email protected]
Sales: [email protected]
4.4 HOURS OF OPERATION
The SpryWare offices are staffed during standard business hours, 7:00 AM CT to 5:00 PM CT on all
market trading business days, as defined each year by the New York Stock Exchange (NYSE). The
2012 through 2014 Holiday Schedules are shown in the following section.
SOP - Master Page 14
4.4.1 Holiday Schedule – NYSE, Canada & London Stock Exchange
NYSE 2012 2013 2014 New Year's Day January 2 January 1 January 1
Martin Luther King, Jr. Day January 16 January 21 January 20
Washington's Birthday* February 20 February 18 February 17
Good Friday April 6 March 29 April 18
Memorial Day May 28 May 27 May 26
Independence Day July 4 July 4 July 4
Labor Day September 3 September 2 September 1
Thanksgiving Day November 22 November 28 7
Christmas December 25 December 25 December 25
TSX 2012 2013 2014 New Year's Day January 2 January 1 January 1
Family Day February 20 February 18 February 17
Good Friday April 6 March 29 April 18
Victoria Day May 21 May 20 May 19
Canada Day July 2 July 1 July 1
Civic Holiday August 6 August 5 August 4
Labour Day September 3 September 2 September 1
Thanksgiving Day October 8 October 14 October 13
Christmas Day December 25 December 25 December 25
Boxing Day December 26 December 26 December 26
LSE 2012 2013
New Year January 3 January 1
Good Friday April 6 March 26
Easter Monday April 9 April 1
May Day May 7 May 6
Spring June 4 March 27
Queen’s Diamond Jubilee June 5
Summer August 27 August 26
Christmas Day (Observed) December 25 December 25
Boxing Day (Observed) December 26 December 26
SOP - Master Page 15
4.5 ORG CHART
SpryWare was founded by Mike Kreutzjans and Dan May in 2004. Both Mike and Dan continue to
play in integral part in the day-to-day running of SpryWare and function as the senior members of
the SpryWare team. A copy of the company Organizational Chart, current as of November 23, 2012,
is shown below. An updated copy of the Org Chart can be found in the Shared Documents section of
the SpryWare SharePoint site.
DirectorsDaniel May & Mike Kreutzjans
VP Sales & MarketingPaul Zinone
Regional Sales DirectorDan Curry
Director of DevelopmentAl Pintoy
Director of Operations & Support
Dan Schwartz
BusinessOperations Manager
Agata Lipinski
Regional Sales DirectorMorgan Roberts
Software Eng.John Dibling
Software Eng.Danny Bordelon
Software Eng.Inna Brodkin
Operations ManagerTerry Guder
Quality Assurance Manager
Ron Lindstrom
Director ofClient Service
Keith Tylke
Director ofEast Coast OPSTony Padovano
Regional Sales DirectorAdam Growick
Operations Support Specialist
Tim Perkin
Operations Support SpecialistDan Kim
4.6 PRODUCTS
4.6.1 The Direct Market Server (DMS)
The SpryWare Direct Market Server (DMS) ™ is a fully managed ticker plant designed to deliver
direct market data with virtually no latency. The DMS™ connects directly to exchange feeds while
normalizing and distributing the data to clients in under 100 microseconds. To achieve this high
throughput, we focus on multithreaded application development making full use of the processing
capacity of modern server class machines.
Low latency market data delivery is the primary goal of the DMS™ but we also include many value
added features. Below is an overview of configurable features included in our MIS ticker plant
technology.
Time Series Database – Can be configured to set time intervals for historic data. Includes
the ability to pick select only data of interest.
Tick Database – By default the MIS stores every trade and quote, but the MIS can be
configured to store any tick database to fit your needs.
Historical Data – Includes 15 years of Equity historical data
SOP - Master Page 16
Historic Data Retrieval and Aggregation – The API we provide with the MIS is highly configurable. For historical data retrieval we use the FAST Protocol with High Speed Indexing.
Database Management – The ticker plant is fully managed and updated in real-time. For example, Corporate action data is updated throughout the trading day rather than after the market close.
Data Validation and Cleansing – The MIS dynamically cleans erroneous data by running each bid, ask and trade from the exchange through a validation algorithm.
Exchange Contract and Roll-Over Management – The MIS automatically handles management of expirations and roll-over information for futures and options and strike prices are dynamically updated.
Fundamental Data – The Fundamental data tables allow for extensive cross referencing including access to instruments by CUSIP, ISIN and SEDOL.
Calculated Fields – The MIS comes preconfigured with real-time calculations like Highs, Lows, and Cumulative Volume and many other important fields that are not provided directly by each exchange.
Data Blending and Linking – DepthPlus allows clients multiple views and sorted levels of the order book. DepthPlus sorts, aggregates and provides all levels of book depth. In addition the client-side API is extremely configurable for a wide assortment of “views’ of the data.
User Authentication and Usage Reporting – Allows for permission controls down to the end-user level.
SOP - Master Page 17
4.6.2 HDS
In addition to SpryWare’s flagship product, The Market Information Server (MIS), SpryWare is
offering HDS. SpryWare HDS is a complete archive of every trade and quote on all major equities,
options and futures.
The HDS archive is stored in the FIX Adapted for STreaming (FAST) format. The FAST
Protocol, sponsored by FIX Protocol Ltd. and co-authored by SpryWare, is the highly compact market
data standard that radically reduces the bandwidth required to transmit ever-increasing real-time
market data message rates. By leveraging field encoding, implicit tagging, and serialization, FAST
reduces bandwidth consumption by up to 90%.
The SpryWare MIS™ processes raw exchange data in various formats and normalizes them using
FAST. This FAST feed is then databased and archived. In applying the protocol to mass storage of
tick-by-tick data, SpryWare is able to make available very large amounts of tick data in files
that are a fraction of the size of our competitors. These large amounts of data are then made
immediately available to our clients and retrieved with ease due to their size.
Complex blackbox and algorithmic trading strategies that require tick-by-tick data combined
with rapidly rising market data volumes make data storage and retrieval a major concern. SpryWare
HDS allows traders to pursue these strategies with ease.
SpryWare HDS Tick Data is currently available for:
OPRA AMEX CBOE Futures CFE
NASDAQ UTP (UTDF & UQDF) Eq SIAC – All regional exchanges CBOE RLC & ITC
NASDAQ OMDF PBOT Indices NYMEX
NASDAQ TDDS Pink Sheets COMEX
NASDAQ BBDS CBOT ITC Kansas City Board of Trade ITC
NASDAQ NIDS CBOE Direct Minneapolis Grain Exchange ITC
NASDAQ MFDS CBOE One Chicago Winnipeg Commodity Exchange
NYSE CBOE Options on Futures Dow Jones Indices
SOP - Master Page 18
5. SERVICE LEVEL AGREEMENT
5.1 SCOPE
This Service Level Agreement (SLA) applies only to Customer’s service performance directly relating
to SpryWare’s Service. The scope of this SLA does not include, under any circumstances, any server
on the Internet, customer premise equipment (“CPE”) or local access service. The interpretation of
this SLA is, at all times, intended to be subject to and consistent with the Spryware Master Licensing
and Service Agreement (“Master Agreement”).
5.2 NOTIFICATION AND CREDITS
Service-impacting conditions can be reported either by Customer or by Spryware. At the time
Customer believes that a service impacting condition has occurred, Customer must initiate a trouble
ticket by contacting Spryware Customer Service in accordance with the method of contact set forth
in the Spryware [Customer Service Guide]. If the Spryware Customer Service believes that a service
impacting condition has occurred, it will open a trouble ticket on Customer’s behalf. Once a ticket
has been opened, Spryware will notify Customer of the situation and initiate diagnostic testing and
fault isolation activities to determine the nature of the service condition. A credit will be applied to
Customer’s first monthly invoice subsequent to Spryware’s determination that a service impacting
condition that lies within the scope of coverage has occurred.
5.3 EXCLUSIONS
The scope of this SLA covers only those events isolated to the areas specified in the Scope section
above. SLA does not apply to scheduled or Customer-requested service interruptions. Outages will
only be measured only in accordance with the sections below. Under no circumstances will any
tests (for example PING tests) performed by Customer, its vendors or partners be recognized by
Spryware as a valid measurable criterion of violation length, quality or type for the purposes of
establishing a service credit hereunder. Failure of Customer to inform Spryware of changes to
Customer’s technical contacts lists may result in the denial of credits under this SLA. The SLA
objectives contained herein apply only to SpryWare’s Customers; they do not apply to customers of
Customer. Under no circumstances will credits be given for outages involving: (a) trouble tickets
associated with new installations (b) trouble tickets erroneously opened by Customer; (c) a circuit
release required by Customer for testing; (d) trouble tickets opened by Customer for service
monitoring purposes only; or (e) trouble tickets related to Customer maintenance, configurations,
negligence, accidents or omissions.; or (f) events beyond SpryWare’s reasonable control, including,
but not limited by, the transmission failure of exchanges and third party information providers.
5.4 CREDIT TYPES
SpryWare offers two types of service credits under this SLA. Proactive credits are automatically
processed by the appropriate Spryware team member, and are credited to Customer’s monthly
invoice following SpryWare’s determination that the event lies within the scope of coverage.
SOP - Master Page 19
Customer is not required to request proactive credits. However, if Customer feels that Spryware has
not applied a credit correctly, Customer is able to request these credits via their Spryware Sales
Team anytime within thirty (30) business days of the resolution of the event in question. Non-
proactive service credits are not applied automatically to Customer’s monthly invoice. Non-
proactive credits must be requested in writing from Customer’s Spryware Sales Team within thirty
(30) business days of the resolution of the event for which the credit is being requested. Spryware
will apply any non-proactive credits due to Customer on Customer’s [monthly invoice] following
their request and SpryWare’s determination that the event lies within the scope of coverage.
5.5 SERVICE AVAILABILITY METRICS
5.5.1 Service Point Problem Resolution Time
Spryware will use commercially reasonable efforts to ensure that eligible outages to an area within
the Service Point and affecting Customer’s specific resource will be resolved as soon as reasonably
possible and, in any event, in less than ten (10) minutes. Failure to meet this objective may make
Customer eligible for a service credit to be applied to Customer’s next monthly invoice for Services.
5.5.1.1 Remedies and Credits
The Service Point Problem Resolution Time Guarantee is determined and calculated on a per-
occurrence basis, commencing upon Spryware Customer Service’s initial awareness of an outage
and ending when the service has been restored. Customer’s service credit is a prorated amount
derived from the length of the service outage and Customer’s Monthly Recurring License Fee
(“MRLF”) for the affected service. The processing of any Customer credit will be done by the
appropriate Spryware team member, who will apply the credit parameters as specified below.
Length of Outage Amount of Credit Credit Type
Less than 10 minutes None
Proactive
10 minutes to 30 minutes 2 days prorated MRLF
31 minutes to 60 minutes 4 days prorated MRLF
61 minutes to 6 hours 6 days prorated MRLF
More than 6 hours 4 weeks prorated MRLF
Second qualifying outage in same calendar month
Two times the amount listed above for the respective second qualifying outage
5.5.2 Request for Reasons for Outage Guarantee
Spryware will use commercially available means to reasonably ensure that Customer will be sent a
formal written Reason for Outage (RFO) on events isolated to SpryWare’s Service Point as quickly as
possible, and in any event less than five (5) business days from the date of Customer’s request.
RFO’s extend beyond SpryWare’s standard notifications given for all events and can contain
additional details regarding an event. This service point specific guarantee excludes backbone
problems and any other services provided by SpryWare’s vendors, partners, or affiliates. Failure to
SOP - Master Page 20
meet this objective may make Customer eligible for a service credit to be applied to Customer’s next
monthly invoice for Services.
5.5.3 De Facto Outages
Spryware recognizes that time is of the essence with respect to the timely transmission of accurate
data, without which the Customer cannot conduct business. Spryware shall endeavor to cure any
material defects in data quality, completeness, and/or transmission speed, within a reasonable
period of time as such defects occur. In the event that such defects occur on a consistent or on-
going basis, Customer has the right to terminate the Master Agreement with thirty (30) days written
notice and without further liability or obligation to the Customer. This section contemplates those
instances where data transmission is incomplete or delayed rather than a complete Outage as
contemplated in Section 4.5.1.1 above.
5.6 APPLICABILITY
a. If Customer experiences four (4) qualifying violations of the same metric as defined in this
SLA on the same service, Customer shall have the right to terminate the affected service in
lieu of service credits for the fourth or final such qualifying event or any other remedy
agreed to by the Parties which is offered for such events and without liability or obligation
except for unpaid Fees and costs relating to the Services rendered through the date of such
termination and all third party termination, disconnection and cancellation fees relating to
such termination, including, but not limited to, CPE gear or Local Telephone/Access
Company's local access charges, provided Customer has notified Spryware of its intent to
terminate in writing within 30 days after the last such outage.
b. Eligibility for any credits is subject to Customer’s account being held current and having no
outstanding balance due. Customer’s total credit in any contract year shall not exceed 60
days fees for the covered service type. Residual credits may not be carried over to
subsequent years.
c. Customer must have contracted for the specific service covered under each provision to
qualify for any credits against those provisions.
d. EACH CREDIT OR TERMINATION RIGHT SHALL BE THE CUSTOMER’S SOLE AND EXCLUSIVE
REMEDY FOR THE CORRESPONDING SERVICE OUTAGE AND SPRYWARE’S FAILURE TO MEET
THE SERVICE OBJECTIVES. ANY DISPUTES ARISING OUT OF OR RELATING TO THIS SLA MUST
BE BROUGHT WITHIN SIX MONTHS OF THE OCCURRENCE OF ANY SUCH DISPUTE.
e. This SLA applies to new Customers who use the Services in accordance with Spryware’s
policies on or after 6-1-2005. This SLA will be applied to all qualifying incidents occurring on
or after 6-1-2005. Spryware reserves the right to change or modify this SLA and will post
such changes or modifications to a web site designated by Spryware and made available to
Customer, provided that any such change or modification shall not adversely affect
Customer’s then-current SLA.
SOP - Master Page 21
6. CUSTOMER SERVICE GUIDE
6.1 OVERVIEW
At SpryWare we understand the urgency and importance of real-time market data. From initial
implementation to the actual use of our technology, you will work closely with our own full-time
professionals who have many years of market data experience and are experts on all of the
SpryWare market data solutions.
Support is divided into the following three functional areas:
Network & Hardware
Database Management
Development and Functionality
6.2 NETWORK MONITORING
SpryWare operates fully managed networks and has secure remote access to all data centers and
servers. An extensive monitoring system is in place that provides statistics and notifications on a
real-time basis. These systems are operational 24/7. Normal system maintenance and updates will
take place outside of regular office hours.
6.3 HOURS OF OPERATION
The SpryWare offices are fully staffed during standard business hours, 8:00 AM CT to 5:00 PM CT on
all market trading business days as defined each year by the New York Stock Exchange (NYSE).
SpryWare provides an additional hour of Customer Support each morning beginning at 7:00AM CT.
SpryWare defines Support Hours as running from 7:00 AM CT to 5:00 PM CT Monday thru Friday on
all market trading days.
System upgrades or other events requiring after-hours support can be accommodated on a case-by-
case basis by SpryWare Support. Instances of after-hours support will need to be arranged in
advance, and are subject to staff availability.
SpryWare staff will also make their best effort to address issues that arise during off-hours. Email is
typically monitored during evening and early morning hours. The SpryWare monitoring system will
email alerts to staff as they are detected. Clients can also email SpryWare at
[email protected] to report issues or make inquiries. SpryWare will follow a “Best Efforts”
policy to address system critical issues reported by the monitoring system and client reported issues
emailed to Support prior to the next business day’s market open.
SOP - Master Page 22
6.4 EMAIL SUPPORT
Issues, requests, comments and suggestions sent to SpryWare via email should be sent to
[email protected]. The SpryWare Support team members are copied on all Support emails.
Emails sent to Support will generate a new case in the SpryWare help desk tool. Each case will be
reviewed and assigned to a qualified team member. The Client will receive an acknowledgement
email from the Support Department telling them a new case has been created. Emails will be
responded to in the timeframes outlined in the Problem Severity, Communication and Response
Definitions section detailed below.
6.5 PHONE SUPPORT
Incoming phone calls to the Chicago Headquarters during the standard Support hours outlined
above are answered by live personal. The Client is instructed to call (312) 922-7779 to be connected
directly to the appropriate Support personnel.
6.6 PROBLEM SEVERITY, COMMUNICATION AND RESPONSE DEFINITIONS
SpryWare defines four levels of problem severity, communication and resolution for client support.
The response times indicated below are not applicable to issues that arise during non-business
hours.
Table 2: Problem Severity, Communication and Response Table
Severity Description Response Time Contact Method
1 Client access is down with no work around available
30 Minutes Phone
2 Client access is available with service feature(s) unavailable with no work around identified
60 Minutes Phone, Email
3 Client access is available with service feature(s) available using a work around
3 Business Days Trouble Ticket
4 Client access is available with all features functioning. Suggestion or inquiry on usage of service.
5 Business Days Trouble Ticket
6.6.1 Definitions
Response Time: The amount of time SpryWare has to provide the first update to the client
after being notified of a problem.
Contact Method: The method of communication to report and monitor a problem until
resolution.
SOP - Master Page 23
7. SPRYWARE WEBSITE
SOP - Master Page 24
8. TOOLSETS
SpryWare relies on a number of tools for day to day productivity. This section provides an overview
of each of these tools. In cases where lengthy documentation is required to detail a specific tool a
brief overview will be provided and more through documentation will be provided in a separate
attachement as noted in that overview.
Toolsets described in this section include:
Active Directory
SaaS Provider Intermedia
o Email – Exchange/Outlook
o SharePoint
o CRM
Salesforce
CVS
Nagios
BGInfo
MISVerify
VPN
8.1 ACTIVE DIRECTORY
SpryWare relies on Microsoft Active Directory and DNS. There are currently three Active Directory
servers, or more commonly known as Infrastructure servers. Each servers runs the following
applications:
Domain Controller
Global Catalog Server
DNS Server
DHCP Server
One of the three servers is located at SpryWare HQ. The other two are Virtual Servers locate at the
Equinix Chicago and NY4 Data centers. The server names are as follows:
SWHQDC01 (SpryWare HQ)
SWILDC02 (Chicago Equinix CH4)
SWNJDC03 (Secaucus Equinix NY4
Additional information on the configuration and use of these resources can be found in the
document: SOP – Active Directory v##.
SOP - Master Page 25
8.2 INTERMEDIA
8.2.1 Overview
SpryWare subscribes to the principals of Software as a Service (SaaS). SaaS is a model of software
deployment whereby a provider licenses an application to customers for use as a service on
demand. Intermedia is such a provider offering SpryWare Email services (Exchange & Outlook),
SharePoint and CRM. All three of these Microsoft products run on servers housed at the Intermedia
data centers. No servers or technical expertise in the operation of these environments is required
by SpryWare. Intermedia is solely responsible for the maintenance of their servers. As a user,
SpryWare works with Intermedia to ensure that full advantage is taken of these offerings. The
following subsections detail these three offerings.
8.2.2 Email – Exchange/Outlook
8.2.2.1 Overview
8.2.2.2 Password Policy
8.2.2.3 Process & Procedures
8.2.2.4 Intermedia Technical Support
8.2.2.5 Create New Mailbox
8.2.2.6 Create New Distribution List
8.2.2.7 Storage Management
8.2.2.8 Archive your Mailbox
8.2.2.9 ActiveSync
8.2.2.10 Blackberry Service
8.2.3 SharePoint
8.2.3.1 Overview
The SpryWare SharePoint Home site is located at the following address:
http://spryware.hostpilot.com
As noted in the sections below certain SpryWare customers and business partners will access the
site using an extended URL, example: BNY will require a “/bny” at the end of the URL.
The following sections detail login procedures and the various sections of the SharePoint site.
8.2.3.2 Initial Access
SpryWare offers two levels of access to the SharePoint site; full access for SpryWare employees and
limited access for select customers and business partners. The full access option allows SpryWare
employees access to the Home site as well as access to select customer/business partner sites. The
SOP - Master Page 26
limited access option allows a SpryWare customer or business partner access to their specific site
only. These access levels are set by the SpryWare SharePoint administrator.
Credentials for individual SpryWare team members are established first through an entry in
SpryWare’s Active Directory, followed by inclusion in the Intermedia Hostpilot Administration page.
Lastly, from the SharePoint site itself, the user needs to be added to the People and Groups section.
Utilizing an Intermedia feature titled UserPilot, the SpryWare Intermedia Adminstrator will tie the
SpryWare users Active Directory credentials to Intermedia so the user will have a single login and
password to gain access to all Intermedia tools.
For the select SpryWare customers and business partners who have SpryWare provided SharePoint
sites, credentials are established through Intermedia Hostpilot as well as the Users and Groups
section in the individual SharePoint site. These crecentials are established by the SpryWare
Intermedia Administrator.
SOP - Master Page 27
The login and password required to gain access to the SharePoint site can be saved so that after
sigining on initially, this information won’t have to be entered again. To do this proceed as follows:
1. Open Internet Explorer
2. Go to the SharePoint Home site. You’ll need to login but this will be the last time.
3. Go to Tools -> Internet Options -> Security -> Trusted Sites -> Click on the Sites button
4. Confirm that the SharePoint URL is listed in the Add this website to the zone: field and
clieck <Add>
5. Click <Close>
6. On the Security tab page, press the Custom level button
7. Scroll down to User Authentication and select Automatic login with current username and
password
Not that these procedures will have to be performed on any computer that is used to gain access to
the SharePoint site; typically a work and home PC.
SOP - Master Page 28
8. Click <OK>
9. Click <Yes>
10. Click <Apply>
11. Select <OK>
8.2.3.3 Home
8.2.3.4 Essex Radez
8.2.3.5 ConvergEx
8.2.3.6 QA
8.2.3.7 Announcements
8.2.3.8 Links
8.2.3.9 Shared Documents
8.2.3.10 Calendar
8.2.3.11 Server Detail
8.2.3.12 Lists
SOP - Master Page 29
8.2.3.13 Phone List
8.2.3.14 Process & Procedures
8.2.3.14.1 Create a new announcement
8.2.3.14.2 Upload a document
8.2.3.14.3 List Updates
8.2.3.14.4 Add/Change/Delete a Calendar Entry
8.2.3.14.5 Server Detail Entry/Modification
8.2.4 CRM
8.2.4.1 Overview
The entire SOP for CRM can be found in the attachment entitled “SOP – CRM User Manual v##”.
This document covers the SpryWare implementation of Microsoft Dynamics CRM Version 4, which
we refer herein to as simply Microsoft CRM or just CRM. The Acronym CRM stands for Customer
Relationship Management. CRM is the tool that will be used to replace Salesforce for Customer
Support ticket processing.
This tool is provided to SpryWare by CRM On Target, which is the leading worldwide provider of
partner-hosted Microsoft Dynamics CRM. TriVenture is the parent company of CRM On Target.
CRM is a highly customizable tool that has been configured to specifically address SpryWare’s
unique Support requirements. Over time this tool will be updated to introduce new features and
functionality.
Initially this tool will be for the processing of Support tickets (Cases). Cases will be created in one of
two ways either automatically via email ([email protected]), or manually from detail received
through a customer call or other customer contact.
This document details the installation, training and Support procedures. An overview of Workflow
processing is detailed. Additionally, a section has been included stressing the importance of keeping
accurate and complete contact and account information.
8.3 SALESFORCE
8.3.1 Overview
8.3.1.1 Home
8.3.1.2 Leads
8.3.1.3 Accounts
8.3.1.4 Contacts
8.3.1.5 Opportunities
8.3.1.6 Reports
SOP - Master Page 30
8.3.1.7 Dashboards
8.3.1.8 Documents
8.3.1.9 SpryWare Products
8.4 CVS
8.4.1 Overview
8.4.2 Process & Procedures
8.4.2.1 Procedure #1
8.4.2.2 Procedure #2
8.4.2.3 Procedure #3
8.5 NAGIOS
Nagios is a monitoring application for both systems and networks that monitors and reports on:
Hosts – computers, switches, and printers
Services – network resources, host resources (CPU load, Memory Usage, etcetera
The entire SOP for Nagios can be found the attachment entitled “SOP – Nagios Manualv ##”.
Monitored elements are evaluated and assigned one of three values:
OK – Element is within standard working parameters
WARNING – Element nearing the border of acceptable parameters
CRITICAL – Element is Non-Responsive, Not Reachable, Not Executing, or Not within
established acceptable parameters.
Notification is performed in numerous methods:
Web Display
Text Message (requires plugin, additional cost)
Audio Alerts
Phone Calls(requires plugin, additional cost)
Email and Web display are currently configured on the system
SOP - Master Page 31
8.6 BGINFO
BGINFO is a utility designed to poll the server for information related to that server’s hardware,
network configuration, operating system and SpryWare version number. This information is
displayed directly on the desktop to allow for easy access to this data. BGINFO collects this
information each time a new user logs into the system thus guaranteeing the most up to date data.
Should the user fail to logoff or remain on the system for an extended period of time BGINFO will
refresh on an hourly basis. The instructions for installing BGINFO are shown below:
Assumptions
The user has logged in as SpryWadm with the /admin switch.
Example: mstsc /v:mis70.spryware.local /admin
Installation Steps:
1. Create the folder: C:\SpryWare\BGinfo on the server in question.
2. Copy the file BgInfo.zip form the ClientApps directory on the D: drive on Prine to the newly
created BGinfo folder on the server in question.
3. Extract the Zip file into the BGinfo folder.
4. Start -> Run -> cmd (enter) 5. Type \spryware\bginfo\info.cmd <HOSTNAME>
(ex. \spryware\bginfo\info.cmd mis70 ) 6. Enter the SpryAdm password when prompted. 7. Close cmd window when finished. 8. Notice the desktop now contains the system information. 9. Log out of the rdp session 10. Log back into the box without using /admin 11. Accept the EULA 12. Notice the desktop now contains the system information. 13. Log out
8.7 MISVERIFY
MISVerify is a utility that checks the status of the Market Information Servers located in the
SpryWare cages at Equinix in Chicago. This utility is scheduled to run via the Task Scheduler utility at
5:00 AM each day.
The MISVerify utility checks for the existence of each server and performs checks against the
following:
Core Tables
o Directory
o Events
o Session
o SIPStats
o Authentication
Specified tables that can include:
SOP - Master Page 32
o EquityPrice
o EquityOptionChain
o FuturePrice
o FutureOptionChain
Symbols that exist in the tables:
o IBM
o /ES 09Z
If MISVerify identifies an error condition, an email will be generated and sent to the Operations
Distribution list with the Subject line reading: MISVerify Errors. This email will include detail on the
error condition(s). This email should be reviewed by Operations in the morning and corrective
action taken to address any reported errors. A MISVerify Errors email will not be sent out if no
errors are detected.
The MISVerify utility and associated files reside on the adm1 Server at Equinix in the directory
c:\MisVerify. In order to add or remove a server from polling, edit the verifyall.cmd file and make
the appropriate changes.
8.8 VPN
SpryWare provides a number of ways for team members to access SpryWare resources remotely
through VPN connectivity. Establish a VPN connection is frequently the first step to being able to
access remote client servers from outside the SpryWare office.
Refer to the document SOP – VPN Guide v## for further information on the setup and use of
SpryWare VPN resources.
8.9 BUGZILLA
Bugzilla is a Web-based general-purpose bug tracker and testing tool used by the SpryWare
Development and Q/A teams. Refer to the document SOP – Bugzilla v## for further information on
the setup and use of this tool.
SOP - Master Page 33
9. DIRECT MARKET SERVER (DMS)
9.1 OVERVIEW
High level stuff. Probably just grab material from Marketing. Talk about the various types of
servers, back end, proxy and eventually MIG.
9.2 TECHNICAL REQUIREMENTS
9.2.1 Hardware
Redundant power
Redundant NICs, market data feeds
9.2.2 Operating System
2008, vs 2003. Standard vs Enterprise. Turn off automatic updates
9.2.3 Network
Detailed in the Installation section… how to configure NICs
Mention Cisco
Two for market data, one for lan and maybe one more for a cross connect
9.3 MIS BUILD PROCEDURES
A MIS can be installed at one of the two SpryWare Hosted Datacenters; Equinix NY4 in Secaucus
New Jersey or Equinix CH4 in Chicago. A MIS can also be installed on a client premises or at a
datacenter of the client’s choosing. Two stand alone documents have been created in order to
document the MIS Build Procedures; one covers the Windows Operating System and one coveres
the Linux based RHEL6 Operating System. The documents are titled:
SOP – Windows MIS Build Procedures v##
SOP – Linux MIS build Procedures v##
The build process starts with documentation, but proper documentation is required throughout.
The SOP also includes a review of the server specification and covers ongoing communications with
the client throughout the process, bringing the Server to a functional state, the physical process of
receiving the server and having it mounted in a cabinet, followed by the installation of power,
network and Keyboard/Video/Mouse (KVM). Also included are the required configuration settings
for the BIOS, steps required to configure the Operating System and setup the network. Lastly, The
SpryWare Server Build section details the steps required to install and configure SpryWare software
on the new server. A Checklist section is provided as a more concise list of steps to be used by a
skilled Operations team member who will be responsible for the actual build process. A Build
Review Checklist is designed to be used by another SpryWare team member (who didn’t do the
original build) as a quality assurance step, to check/verify the work of the person who did the
original build.
SOP - Master Page 34
9.4 MIS.XML
This section documents the the three types of MIS.XML files, the Client Side, Data Server Side and
Proxy. The Client Side and Proxy configureation is documented in full detail with all applicable
settings. A brief description of the Data Server Side MIS.XML is also included. A more detailed
review is beyond the scope of this document.
The MIS.XML resides in the \SpryWare\Bin directory and is responsible for the overall configuration
of the client computer or server.
SOP - Master Page 35
9.4.1 Client Side
Command Description <MisConfig> Every MIS.XML must start and end with this.
<Directory> Section for basic client server settings. <Remote>[True/False]</Remote> This setting should always be set to the default of True.
<Login>Jsmith</Login> Login
<Password>Password</Password> Password <Domain>Spryware</Domain> Domain
<DataDirectory>C:\Users\Jsmith\Data</DataDirectory> Data directory field only required to redirect the location of the temporary files. The default is the TEMP directory
</Directory>
<Client> Method of connection to the server.
<RemoteType>[Socket|LBM]</RemoteType> Default is Socket. LBM (Latency Busters Messaging) not yet supported. </Client>
<Socket>
<TableSocket> host[:port]</TableSocket> Server name or IP and port no. Default port number is 80
<EnableDebugLogging>[Yes|No]</EnableDebugLogging> Enables logging. Yes=On, No=Off
<NoDelay>[On|Off]</NoDelay> Turns Nagle On/Off. NoDelay ON means Nagle is Disabled, NoDelay OFF meands Nagle is Enabled
<TCPWindowSize>1048576</TCPWindowSize> Specifically sets TCPWindowSize. The default is left to the O/S.
<BaseTimeout>30</BaseTimeout> Controls Base Query timeout representing the number of seconds to wait for a response from the server before reconnecting. Default is 2
</Socket>
<LogManager> Application required for logging
<LogDirectory>C:\Users\Jsmith\Logs</LogDirectory> Directory log files will be written to. Default is C:\SpryWare\Data
</LogManager>
<MisGiver>
<AutoClose>[On|Off]</AutoClose> Close MisGiver after application utilizing it is stopped. On=Close, Off=Don’t Close.
<Daemon>[Yes|No]</Daemon> Yes=Launch MisGiver as a Helper as required No=Don’t launch
<UnloadDelay>2</UnloadDelay> Delay in seconds before MISGiver closes after corresponding application closes. Default is 5
</MisGiver>
<BufferQueueSize>
<All>500</All> Number of messages. Default is 0 (Unlimited)
</BufferQueueSize>
<PricePlus> Settings specifically for PricePlus example program <Size>10000</Size> Default table size for local cache. Default is 10000
<CalcBBOSize>[Yes|No]</CalcBBOSize> Control National BBO Calculation. Default is No
<GrowthSize>5000</GrowthSize> Grow Size increments in number of records <MultiThreaded>[On|Off]</MultiThreaded> Run MultiThread. On=Run, Off=Don’t Run. Default is On
</PricePlus>
<MISRTDServer> Excel Real Time Data Section
<Debug>[On|Off]</Debug> Turn debugging on or off. Default is Off <TableSocket>host[:port]</TableSocket> Line typically not included. Default to <Socket> section
</MISRTDServer>
</MisConfig> Every MIS.XML must end with this.
SOP - Master Page 36
9.4.2 Server Side (Data Server)
The Server Side MIS.XML resides in the \SpryWare\Bin directory and among other things, dictates
which incoming exchange feeds the server will accept. As indicated at the beginning of this section,
comprehensive documentation of this file is beyond the scope of this document due to the length
and complexity of each setting. For a new MIS install, copy a MIS.XML from a similarily configured
MIS and make any required changes specific to the server in question. Consult SpryWare
Operations and/or Development for details associated with specific configuration settings.
SOP - Master Page 37
9.4.3 Server Side (Proxy)
The MIS.XML detailed below is from a Proxy server attached to two back end servers. In this
example the back end servers are MIS50 & MIS51.
Command Description <MisConfig> Every MIS.XML must start and end with this.
<Directory> Section for basic client server settings.
<Remote>[True/False]</Remote> This setting should always be set to the default of True. <Login>Jsmith</Login> Login
<Password>Password</Password> Password
<Domain>Spryware</Domain> Domain
<DataDirectory>C:\Users\Jsmith\Data</DataDirectory> Data directory field only required to redirect the location of the temporary files. The default is the TEMP directory
</Directory>
<Client> Method of connection to the server.
<RemoteType>[Socket|LBM]</RemoteType> Default is Socket. LBM (Latency Busters Messaging) not yet supported.
</Client>
<Socket>
<TableSocket>mis50:80</TableSocket> Server name and port no. Default port number is 80
<TableSocket>mis51:80</TableSocket> Typically a second (or more) servers are included. <EnableDebugLogging>true</EnableDebugLogging> Enables logging. Ture=On, False=Off
</Socket>
<MisGiver> Application required for logging
<AutoClose>Off</AutoClose> Close MisGiver after application utilizing it is stopped. On=Close, Off=Don’t Close.
<Daemon>yes</Daemon> Yes=Launch MisGiver as a Helper as required No=Don’t launch
<ConnectOnStartup>true</ConnectOnStartup> Connect to back end servers on statup.
<SharedInterest>Off</SharedInterest> <InterestListSize>1000</InterestListSize>
<VerifyBBOInterest>Off</VerifyBBOInterest> <ForwardOptionOnRoot>Off</ForwardOptionOnRoot>
</MisGiver>
<QuoteServer>
<DataDictionaryType>ZLIB</DataDictionaryType> <StreamDictionaryType>FAST</StreamDictionaryType>
<MaxQueueCount>200000</MaxQueueCount>
<Priority>High</Priority> <MaxPacketsPerBuffer>1000</MaxPacketsPerBuffer>
<RejectEntitlements>Off</RejectEntitlements>
</QuoteServer>
<LogManager> <LogDirectory>c:\spryware\data</LogDirectory>
</LogManager>
SOP - Master Page 38
Command Description <Fast>
<Trade>
<Last>Price</Last> <LastSize>UInt32</LastSize>
<TradeCondition>TradeCondition</TradeCondition>
<Volume>UInt32</Volume> <ParticipantTotalVolume>UInt32</ParticipantTotalVolume>
<Open>Price</Open>
<High>Price</High> <Low>Price</Low>
<FinancialStatus>UInt32</FinancialStatus>
<SecurityStatus>UInt32</SecurityStatus> <CancelSequence>UInt32</CancelSequence>
<Close>Price</Close> <OfficialOpeningPrice>Price</OfficialOpeningPrice>
<OfficialClosingPrice>Price</OfficialClosingPrice>
<VolumeExpanded>UInt64</VolumeExpanded> <ExtendedPrice>Price</ExtendedPrice>
<ExtendedPriceSize>UInt32</ExtendedPriceSize>
<Underlying>Sku</Underlying> <NetChange>Value</NetChange>
<PercentChange>Value</PercentChange>
<SellerDays>StringZ</SellerDays> <TradeCondition2>TradeCondition</TradeCondition2>
<TradeCondition3>TradeCondition</TradeCondition3> <TradeCondition4>TradeCondition</TradeCondition4>
<TradeDate>Date</TradeDate>
<MarketVWAP>Price</MarketVWAP> <VolumeWeightedAveragePrice>Price</VolumeWeightedAveragePrice>
<TotalValue>Int64</TotalValue>
</Trade> </Fast>
</MisConfig> Every MIS.XML must end with this.
SOP - Master Page 39
9.5 MIS DECOMMISSION PROCEDURES
Periodically a server will need to be removed from a SpryWare hosted data center. Reasons will
include, but not be limited to:
Client Replacing Older Hardware
Client no longer a SpryWare Customer
The following steps outline the general procedures required to be followed in order to
decommission a server. Each decommission is unique and procedures may vary. No decommission
procedures should be followed without a formal CRM Work Order.
1. Confirm with the client the Decommission schedule 2. Have all Nagios notifications for the server turned off 3. Have server removed from MISVerify script checks 4. Shut Down the Switch Ports 5. Stop SpryWare Services 6. Uninstall SpryWare software 7. Remove SpryWare services 8. Save off the mis.xml to your local machine 9. Delete install directory 10. Delete data directory 11. Delete any Spryware Desktop shortcuts 12. Get rid of random Spryware related files in c:\temp 13. Delete the Utils folder 14. Uninstall Nagios then delete the Nagios folder 15. Uninstall NTP then delete NTP folder 16. Remove SNMP Traps & Security 17. Edit the host file and take out all mappings except for ‘local’ 18. Remove IP Addresses from Market Data Interfaces, enable any disabled ports 19. Rename SpryAdm user to Administrator 20. Reset the password for Administrator to Password1 21. Reset DRAC Username & Passowrd to root and remove IP config 22. Power Down the Server 23. Remove port assignments from the PDUs 24. Remove KVM from the Dell Remote Access Controller on ADM1 25. SharePoint
a. Change the Status from Active to Decommissioned b. Update the Journal field stating the reason for the Decommission and any other
pertinent information. 26. Schedule/Coordinate Removal of the Server
a. On-site hand over b. SmartHands
27. Remove all unused cabling 28. CRM Work Order
a. Updated throughout the decommission steps b. Close out ticket at the conclusion of the decommission
SOP - Master Page 40
9.6 UTILITIES
9.6.1 Tableviewer
9.6.2 MDUMP
9.6.3 Tail
9.6.4 MISTOP/MISTART
9.6.5 Snapshot
SOP - Master Page 41
9.7 PROCESS & PROCEDURES
9.7.1 SpryWare Upgrade
9.7.2 Data Editing
At some point, either a Spryware customer or Ops member will identify either missing or erroneous
data on a server. The dicussion below will step through the process of editing data on a specific
server and, if need be, populating that change out to all the servers.
9.7.2.1 Procedures
Data is edited following these procedures:
9.7.2.1.1 Verify the data in question.
Typical problems will be with the EquityPrice table and either the open, high, low, or close
values. Check other sources such as Esignal, Yahoo Finance, or Google Finance and come to a
consensus on what the correct value should be.
9.7.2.1.2 Applying Corrections
Open tableviewer on the server in question, go to the specific table, find the symbol and the
turn on ‘Enable Editing’.
SOP - Master Page 42
This will allow you to then select the specific field, double click, and manually type in a value.
This will update that value permanently on that server. Be sure to turn of ‘Enable Editing’ when
finished.
9.7.2.1.3 Broadcast Changes
ADM1.spryware.com is the master server that will propogate changes out to all servers. This
requires the creation of a .tbl file that will be processed by the server and pushed to other
servers. There are several ways that this can be done, but below is one way of creating the file.
Using snapshot will pull the data required to produce a file for the whole system. First, the field
headers seen in a tableViewer EquityPrice table need to be added to a file. The following
command will get those:
Snapshot /t:EquityPrice /s:XXXX >filename.csv
Example:
SOP - Master Page 43
By opening up the created csv in notepad, you can see those headers. Next, pump the data into that
file using the following command:
Snapshot /t:EquityPrice | findstr “symbol” >>filename.csv
Example:
Next, bring up the data file in Excel and remove all the data that you do not need. In example, we
will only be using the ‘close’ data, so we will remove everything else. Save this new file as
EquityPrice.csv. Finally, open that file in Notepad and then save it as EquityPrice.tbl.
9.7.3 Operating System Patches
9.7.4 Conflation
9.7.5 Arbitration
SOP - Master Page 44
9.7.6 Front End/Back End/Proxy/MIG
9.7.7 MIS Redundancy/Fail Over
9.7.8 Server Detail/BGInfo
9.7.9 Backup & Restore Procedures
9.7.10 External RAID Arrays
On select servers withing the SpryWare Equinix Data Centers, CH4 & NY4, SpryWare holds large
amounts of data on external storage RAID arrays. At this time six servers have these RAID arrays
attached to them. The table below lists the server, the RAID array model and capacity:
Server RAID Array Model Capacity
ADM10 (CH4) MD1000 #1 MD1000 #2
26TB 14TB
ADM11 (NY4) MD1000 26TB
MIS74 MD1000 26TB
MIS69 MD1220 3TB
NJMIS21 MD1000 26TB
NJMIS39 MD1220 3TB
9.7.11 Hardware Upgrades
9.7.12 Hardware Failures
SOP - Master Page 45
9.8 TROUBLESHOOTING GUIDE
Procedures for checking the health of the system
Tableviewer
MDUMP
Snapshot
MIS.LOG & Tail
Check status of the services
Copytable
Network – PING, Tracert, etc.
Grep
Access via RDP
Access via keyboard
Access to PDUs
SOP - Master Page 46
9.8.1 Client Side Logging
Enabling logging on the client system will provide detailed information, specifically how much data is
being consumed by the client and how well or poorly data is being queued, buffered and processed.
In order to enable logging, the Logmanager service will need to be installed and the MIS.XML
configuration file will need to be updated.
9.8.1.1 Installation Procedures
As a prerequisite the SpryWare MIS32 bit SDK must already be loaded on the client computer in
question. For instructions on how to load the SDK, refer to the SpryWare SDK Installation
Procedures.
9.8.1.1.1 Logmanager Service Procedures:
1. From a shell (command prompt, ‘cmd.exe’) CD to the SpryWare installation directory. By
default this is “c:\spryware\bin”
2. Execute “logmanager –install” to install the service. An OpenService failed message will
appear but the service should install successfully as shown below:
SOP - Master Page 47
9.8.1.1.2 MIS.XML Update Steps:
1. Open the MIS.XML file for editing.
2. In the “<Socket>” section add a tag called EnableDebugLogging and set it to “on”. An
example <Socket> section is shown below:
<Socket>
<TableSocket>mis66.spryware.com:80</TableSocket>
< EnableDebugLogging>yes</EnableDebugLogging>
<Socket>
3. Create a section in the file called Logmanager. This should be at the same level as <Socket>,
<Directory>, etc. That is, it is not embedded as a subsection but contained within the scope
of <MISConfig> (the top layer.) Add a tag to this section called LogDirectory and set it to a
directory path. This is the file system location that the logmanager service will create logs
in. An example is shown below:
<LogManager>
<LogDirectory>c:\spryware\data</LogDirectory>
<LogManager>
Once these configurations are in place, merely starting the “SpryWare Logmanager Service” via the
Services MSC, or at the shell with “net start logmanager”, will allow events to be logged. No
changes need to be made to the application. The EnableDebugLogging value will write an entry to
the MIS.LOG file when an error or failover occurs.
SOP - Master Page 48
10. INFRASTRUCTURE
In addition to the Market Information Server (MIS) described above, we have several different types
of other servers utilized in the various SpryWare hosted data centers. An overview of each of these
servers is provided in the following sections.
10.1 ADM
10.2 ESXI
Refer to the document entitled “Standard Operating Procedures – ESXi Overview” for more details.
10.3 NAS
10.4 DELL MD1000
SOP - Master Page 49
11. MARKET DATA
11.1 OVERVIEW
The SpryWare MIS is capable of reading a wide variety of domestic and foreign market data feeds.
New feeds are being added or updated all the time. A listing of all currently supported market data
feeds is provided in a sub section shown below.
For all servers residing at one of SpryWare’s hosted data centers, market data is supplied through
SpryWare’s business partner, Essex Radex Company (ERCO). Incoming market data feeds are the
responsibility of the client for all servers that reside outside of one of SpryWare’s hosted data
centers, i.e., client site or non-SpryWare hosted data center.
An Exchange Feed distribution diagram outlining all market data feeds available at SpryWare’s
hosted data centers is available in the Operation section of the SpryWare SharePoint site.
The content and distribution mechanism associated with market data feeds is not static, updates
occur on a regular basis. This may include content changes, hours of operation changes, bandwidth
upgrades, network requirements and various other solutions designed to address expansion
requirements. As such, SpryWare continuously monitors market data change notices that come in
from a variety of sources. Each proposed change outlines the requirements and sets a schedule that
may include test periods and a cut over period. The transition of the new change into Production
may consist of a measured roll out or hot cut at a specific time and date. The handling of these
market data feed changes is detailed in this section.
11.2 SHAREPOINT MARKET DATA CALENDAR
Included in the SpryWare SharePoint Site Home page is a link to a list titled Market Data Calendar.
This list is used to document all upcoming market data feed changes and the required procedures
that need to be followed to ensure a successful roll out. A screen shot of a sample Market Data
Calendar entry is shown below.
SpryWare monitors a number of resources to identify and evaluate changes to market data feeds.
These changes are reviewed by the SpryWare Development team and updated in the SharePoint
Market Data Calendar section. Updates requiring development work are added to the Development
and QA Sections of SharePoint.
SOP - Master Page 50
11.3 MARKET DATA CALENDAR SOURCES
A variety of resources are available to notify market data suppliers and recipients of upcoming
changes. This section details the various resources, including how to subscribe to the update
information, how & when updates become available and the procedures SpryWare follows to
monitor, document and act upon the changes.
11.3.1 NASDAQ OMX
News Alerts are made available on the NASDAQ OMX website located at:
http://www.nasdaqtrader.com/HomePage.aspx
Click on the News Alerts tab to view the current headlines.
These news alerts can be emailed by filling out the Enrollment Form supplied by The NASDAQ OMX
Group. This form is located at:
https://www.nasdaqtrader.com/EASP/TraderEASP.aspx?id=NewsSignup
Create an account and select the alerts to be received.
SOP - Master Page 51
11.3.2 Clearnet Calendar
11.3.3 Interactive Data Corporation (IDC) Developer Notices
Stand alone SOP
SOP - Master Page 52
12. SPRYWARE HOSTED DATA CENTERS
12.1 OVERVIEW
SpryWare allows for flexibility in how The Market Information Server is installed and configured. It
starts with the market data. We can work with existing direct data feeds if a customer is already a
subscriber, or we can provide the data. The same goes for the actual data center location. As
outlined below, we can install in either the customer’s data center or set up a new server in one of
our locations. This section specific to spryware hosted data center…
For customers who are not interested in managing various data lines and networking in a data
center, SpryWare Hosted offers an ideal solution. Take advantage of existing infrastructure and
economies of scale and co-locate your SpryWare MIS Servers with SpryWare at either or both of our
data centers in New York and Chicago. We are located at the very source of the exchange feeds to
eliminate any latency produced by geography.
Each data center leads the industry for physical security, power availability, infrastructure flexibility
and customer support, exceeding the standards set by global Internet brands and leading
enterprises. Each data center is designed to support and protect mission-critical Internet
operations.
12.2 REDUNDANCY
SpryWare integrates redundancy in both the Hosted Data Center and Client premise equipment. All
Market Information Servers are built to a specifications that dictate multiple power supplies,
multiple Network Cards and hard disks.
Incoming Market Data feeds at the Hosted Data Centers are … two vlans multiple nics. This is also
encouraged for CPE devices.
In addition to these server and network specific redundancy options … configured for multiple front
end and back end servers…
12.3 DISASTER RECOVERY
SpryWare has implemented a number of policies to address the issue of disaster recovery. System
monitoring and diagnostics are implemented to identify problems before they reach a critical stage.
For situations where a Severity 1 or 2 issue does occur, these monitoring and diagnostic tools will
notify SpryWare immediately so that steps can be taken to restore service.
12.4 SUPPORT
Qualified technicians are on-site 24 hours a day, 365 days a year at the Chicago Data Center to
perform routine and emergency maintenance/installation procedures. In New York, timely on-site
support is available during each market trading day from 8:00 AM CT to 5:00PM CT. After hours
SOP - Master Page 53
support in New York is available on a “Best Efforts” basis as described in Customer Service Guide
section of this document.
12.5 CUSTOMER CARE AREA
The Chicago Data Center offers a variety of support services. To allow efficient and speedy
customer access and to accommodate the demands of lengthy system/software installations,
customer care areas include:
Individual workstations with telephone and Internet connections
Anti-static equipment staging area
Kitchen, relaxation area, video game room, shower facilities
Shared conference rooms
Secure loading docks to facilitate equipment delivery or shipping
12.6 SITE ACCESS
Three types of site access include Direct Access by a SpryWare employee, Smart Hands & Remote
Access. Discuss Equinix Admin access and who can visit. Tony to provide the rules for gaining access
to SIAC.
12.7 DATA CENTER INFRASTRUCTURE
12.7.1 Overview
All elements of each data center – building shell, exterior, floors and roof – meet or surpass local
building codes and standards.
12.7.2 Physical Security
Multi-level physical security features and a rigidly controlled operating environment protect
valuable customer assets and operations. All areas of each data center are monitored and recorded
using CCTV, and all access points are controlled. Additionally:
Each facility is manned by onsite security on a 24x365 basis.
No keys required: In Chicago all doors, including cages, are secured with biometric hand
geometry readers. In New York a security guard is required as an escort directly to the
appropriate cabinet.
Silent alarm and automatic notification of appropriate law enforcement officials protect all
exterior entrances.
To enter a facility, a customer must present a valid picture I.D. Only pictured government-
issued forms of identification will be accepted.
All equipment checked upon arrival.
Shipping and receiving area walled from collocation areas.
12.7.3 Network Operations Center (NOC)
SOP - Master Page 54
The environmental components of each data center; Security, Power, HVAC, Fire Protection, are
monitored at the Hosted Data Centers Network Operations Center (NOC). Each NOC is staffed
7/24/365 to ensure seamless operation of their respective facilities.
All Market Information Servers are monitored 24 hours per day, 7 days per week by the SpryWare
Operations department. This includes both servers housed at one of SpryWare’s Hosted Data
Centers and Customer Premise Equipment.
12.7.4 Power Systems & Distributed Redundancy
Highly reliable power is imperative for critical customer operations. The entire electrical system has
built-in redundancy to guarantee continuous operation. The overall system is N+1 redundant,
including each component within the parallel electrical systems.
AC and DC raceways with 2N distribution
AC power delivery via distributed redundant UPS systems
Batteries with at least 7 minutes full load operation (diesel engine generators take roughly 8
seconds to synchronize and assume load); 48 hours worth of generator fuel, contracts with
multiple fuel providers.
Isolation K factor transformers used for 480 volt UPS to 208/120 volt. K factor of K20; 80
degrees Centigrade rise; cobber winding, DC system fuse protection; -48 volt delivery via
fuse panels.
12.7.5 Customer Power
Power systems are designed to meet customers’ diverse needs. Typically each cabinet receives two
20Amp Circuits Primary and two Secondary. AC power systems are capable of delivering both 120v
AC and 208v AC power in a variety of amperage configurations.
12.7.6 Network (Internet & Carrier Detail)
Each of SpryWare’s Hosted Data centers are network neutral utilizing multiple Tier 1 network
providers including…
12.7.7 Environmental Controls
To provide optimum conditions for equipment operation and minimize downtime due to equipment
failure, the HVAC system provides appropriate airflow, temperature and humidity. Redundancy
features provide additional protection for customer operations.
A Power Distribution Unit (PDU) is a highly reliable; multiple outlet power strip designed to deliver
conditioned power to mission-critical server, networking and support equipment. These units are
used throughout SpryWare’s Hosted Data Centers used in conjunction with Uninterruptible Power
Supplies (UPSs).
SpryWare utilizes metered PDUs, which allow for monitoring the total amount of current, in amps,
that flow through the PDU. These multi-outlet devices provide sufficient outlets for the many
SOP - Master Page 55
devices that are commonly installed into a rack enclosure. The metered aspect of the PDU helps
ensure that the PDU is not overloaded past the recommended rating of the unit.
Each cabinet at the SpryWare Hosted Data Centers is typically equipped with two to four PDU
devices. For equipment that comes configured with multiple power supplies, two PDUs will be
employed for that particular piece of equipment, thus providing a greater level of redundancy.
Each PDU is accessible via an in-house, or SpryWare VPN enabled, web based interface. The Server
Detail section of the SpryWare SharePoint site lists the PDU number and associated port(s) for each
piece of equipment housed in the data center.
12.7.8 Fire Protection
SpryWare Hosted Data Centers are protected with a dual-alarmed, dualinterlock multi- zoned, dry-
pipe, water-based fire suppression system armed with sensory mechanisms (HSSD) to sample the air
and give alarms prior to pressurization. Production area fire suppression is provided by a
multizoned, pre-action, dry-pipe system. In order for the system to trip, multiple cross-linked events
must occur. These include detection by ceiling mounted smoke-heads and smoke “sniffers” located
throughout the facility. Lastly a sprinkler head must trip in order for the dry- pipe system to activate.
This requires a temperature of 140 degrees F at the head location. Fire suppression is localized at
the event point only.”
12.7.9 Flood Control
The Chicago Data Center offers a number of additional features. The New York facility may offer
similar features, however, document research is still being performed to determine this.
The facility is built above sea-level with no basements. Conduits are tightly sealed, with moisture
barriers on exterior walls. There are dedicated pump rooms and moisture detection sensors are in
place. Drainage/evacuation systems are implemented.
12.7.10 Earthquake Preparedness
The Chicago Data Center offers… Location-specific seismic compliance. Structural systems meet or
exceed seismic design requirements of local building codes for lateral seismic design forces. In
addition, equipment and nonstructural components, including cabinets, are anchored and braced in
accordance with the requirements of the 1997 Uniform Building Code.
SOP - Master Page 56
12.8 SPRYWARE INFRASTRUCTURE
12.8.1 Overview
12.8.2 PDU
A Power Distribution Unit (PDU) is a highly reliable; multiple outlet power strip designed to deliver
conditioned power to mission-critical server, networking and support equipment. These units are
used throughout SpryWare’s Hosted Data Centers used in conjunction with Uninterruptible Power
Supplies (UPSs).
SpryWare utilizes metered PDUs, which allow for monitoring the total amount of current, in amps,
that flow through the PDU. These multi-outlet devices provide sufficient outlets for the many
devices that are commonly installed into a rack enclosure. The metered aspect of the PDU helps
ensure that the PDU is not overloaded past the recommended rating of the unit.
Each cabinet at the SpryWare Hosted Data Centers is typically equipped with two to four PDU
devices. For equipment that comes configured with multiple power supplies, two PDUs will be
employed for that particular piece of equipment, thus providing a greater level of redundancy.
Each PDU is accessible via an in-house, or SpryWare VPN enabled, web based interface. The Server
Detail section of the SpryWare SharePoint site lists the PDU number and associated port(s) for each
piece of equipment housed in the data center.
12.8.3 KVM
Keyboard/Video/Mouse (KVM) Digital Switches are in place for access to all the SpryWare servers
housed at the Hosted Data Centers. These switches can be accessed remotely through a web based
interface. The procedures for accessing the KVMs at SIAC and at Equinix differ and are documented
in the sections specific to each Data Center.
12.8.4 Dell DRAC
12.8.5 Network
SpryWare employs Cisco 4948 and 3750 switches throughout their network.
12.8.6 Cabling
12.8.7 Monitoring
12.9 POLICIES & PROCEDURES
12.9.1 New Cabinet Build Out
12.9.1.1 Site Access & Cabinet Combination
Access to SIAC/Equinix. Access to the cabinet. Access to get into the cabinet.
12.9.1.2 Networking
12.9.1.3 PDU
SOP - Master Page 57
Each cabinet at the Equinix and SIAC Data Centers is equipped with two to four PDUs. Each PDU can
be accessed remotely so that outlets supply power to a MIS can be turned on and off. During MIS
initial configuration or during upgrades the outlets supplying power to the MIS should be turned off
while the work is being performed.
At times a MIS may lock up and become unreachable through RDP or the KVM. In situations like this
power can be turned off to the MIS thus shutting it down. When power is reapplied the MIS is
configured to perform and auto restart thus allowing access to the server once again.
12.9.2 Cable a New MIS
SOP - Master Page 58
12.9.3 PDU Outlet Configuration
In order to change the state (On/Off) of an outlet or outlet(s)apply proceed as fellows:
1. Identify the MIS requiring power to be applied or removed.
2. Login to the SpryWare SharePoint site.
3. Click on Server Detail
4. Select the view Remote Access
5. Scroll down to the MIS in question. In the right hand columns right click on the
corresponding link PDUxx Click Here, where xx represents the PDU number. Select Open in
New Tab
6. A User Login window will appear. Enter the Login & PW as documented in the SpryWare tab
of the LetMeIn Spreadsheet found in the Operations folder on the SpryWare SharePoint site.
The Sentry Remote Power Manager window will appear.
SOP - Master Page 59
7. In the Control Action column select the ports associated to the MIS in question and select
On or Off depending upon the desired result. The power state will NOT be changed once
these selections are made. This will occur in the next step.
8. Once the appropriate On/Off selections have been made click on Apply at the bottom of the
screen as shown below.
9. Click on Logout once complete.
12.9.4 Cisco 4948 Installation & Configuration
12.9.5 Cisco 4948 Port Configuration
12.9.6 Dell DRAC
12.9.7 Chicago Equinix – Specific Procedures
12.9.7.1 Shipment
12.9.7.2 Cross Connect
SOP - Master Page 60
12.9.7.3 Smart Hands
12.9.7.4 KVM Access
Note: Add detail on Resync to discover new servers.
RDP is the preferable way to access a server; it is much quicker then connecting remotely into the
KVM. Typically KVM access is only required when configuring a server’s BIOS, RDP is unavailable due
to a network configuration issue or perhaps if new drivers are being installed.
Currently three Dell Keyboard/Video/Mouse (KVM) Digital Switches are in place for access to all the
SpryWare servers housed at the Chicago Equinix data center. Every Chicago Equinix server can be
access via remote KVM by connecting to SpryWare Server ADM1 and accessing the Dell Remote
Console Switch Interface. These procedures are outlined below:
1. RDP into adm1.spryware.com
2. From ADM1 locate the Dell Remote Console Switch Software icon on the desktop and load it.
3. A window will appear. Click on the Serves tab at the top of the window to view all servers
available through the Chicago Equinix Dell KVMs. This window is shown below:
SOP - Master Page 61
4. Locate the server in question and double click. A login window will appear. Enter the Login &
PW as documented in the SpryWare tab of the LetMeIn Spreadsheet found in the Operations
folder on the SpryWare SharePoint site. KVM access to the server will now be established as
shown in the example illustration below:
5. At the top of the screen click on the thumbtack icon to keep the menu in place. Click on the icon
that shows the number 1 and a mouse pointer in order to combine the two mouse pointers
shown. In order to separate the mouse pointers click on <F10>.
Note: Click on the Macros menu option to gain access to frequently used keystroke
combinations.
SOP - Master Page 62
6. At this point the Server can be accessed by selecting the CTRL + ALT + DELETE Macro selection
and entering the login and password.
12.9.7.5 Facility Tours
Any SpryWare employee with Admin Access to Chicago Equinix can provide a facility tour at any
time with no advanced notice. Guests must be 18 years of age or older and have a valid form of
identifidation to be granted access.
12.9.8 New York SIAC – Specific Procedures
12.9.8.1 Shipment
12.9.8.2 Cross Connect
12.9.8.3 KVM Access
RDP is the preferable way to access a server; it is much quicker then connecting remotely into the
KVM. Typically KVM access is only required when configuring a server’s BIOS, RDP is unavailable due
to a network configuration issue or perhaps if new drivers are being installed.
A single Keyboard/Video/Mouse (KVM) Digital Switch is in place for access to all the SpryWare
servers housed at the SIAC data center. This switch can be accessed remotely through a web based
interface.
Prerequisites: When accessing the KVM from behind a firewall, the following ports must
be enabled: 80, 443, 2068 & 8192.
Java must be installed on the local computer. Without it the user can gain
access to the web page and login. However, a connection to the remote
computer cannot be made.
Follow the procedures outlined below to gain access to the KVM:
7. From a web browser proceed to:
https://198.140.45.161/
8. A Security Alert window may appear. Click <Yes> when asked “Do you want to proceed?”
9. A Web page may appear with the top line reading: There is a problem with the website’s
security certificate. Click on: Continue to this website (not recommended)
10. A User Login window will appear. Enter the Username and Password. (Contact Technical
Operations team for login credentials.) Click on Login.
11. An Avocent AutoView webpage will appear with a series of selections listed down the left hand
pane. Click on Target Devices. A window similar to the image shown below will appear.
SOP - Master Page 63
12. Click on the desired server as listed in the Name column. The Overview window will appear as
shown below with the desired server name shown in the Name field.
SOP - Master Page 64
13. Click on KVM Session to gain KVM access to the desired server. A window representing the
server’s desktop will appear as shown below.
14. A KVM session has now been established to the server. Note the following:
a. To issue a CTRL + ALT + DEL command click on Macros selection at the top of the
window and choose the Ctrl-Alt-Del command.
b. The mouse pointer may appear twice some distance apart from one another. One
mouse pointer represents the mouse on the local computer with the second
representing the remote computer. Click on the mouse pointer icon in the menu
at the top of the screen to join the two. In doing this the mouse can’t leave the session
window; i.e., can’t access the local computer. Additionally with the mouse pointer
joined together the menu at the top of the screen can’t be accessed. To separate these
hit the <F10> key.
15. To exit the session click on File -> Exit or click the icon in the top right hand corner of
the window.
SOP - Master Page 65
13. CLIENT SUPPORT
13.1 OVERVIEW
Introduction of New Client Procedures
Introduction of Existing Client Change Management
Remote Access Procedures
Documentation – Server Detail, Visio, RCA
Status Meetings
Communications Plan
Monitoring – Internal & External
Proactive approach toward hardware & software & network
Account Manager – Describe various roles we play Demo/Technical Lead,
Onboarding/Implementation Manager, Salesperson/Account Manager Status reports for the
Director Meeting.
SpryWare follows a detailed workflow when
SpryWare porovides flexibility in how The Market Information Server is installed and configured. It
starts with the market data. SpryWare works with the client’s existing direct data feeds if they’re
already a subscriber or can provide the data for the client. The same goes for the actual data center
location. As outlined below SpryWare can install in either the client’s data center or set the client’s
server up in one of our locations. This section specific to client premises installed MIS…
For customers who have sourced data directly from the exchanges and various other data sources,
this solution provides the ultimate in low-latency market data processing. Providing the customer
complete control of the network architecture and all inbound market data lines, the SpryWare MIS
appliance is installed directly in the client’s datacenter.
13.2 SYSTEM MONITORING
SOP - Master Page 66
13.3 REMOTE ACCESS
Due to network and security policies, accessing client servers remotely requires procedures unique
to each client. The procedures to be followed to gain access to servers housed at remote client sites
are documented in the attachment entitled “SOP – Client Remote Access Procedures v##”.
These procedures do not cover servers located at the SpryWare Hosted Datacenters located at,
Equinix Chicago (CH2 & CH4) and the New York (NY4) Equinix facilities. To access these servers a
VPN connection to needs to be established (SOP – VPN Guide v##) at which point each server is
available through RDP on your local computer.
Regarding User names and passwords, a standard set of credentials has been created for access to
all servers located at a SpryWare Hosted Datacenter. These credentials will also work on a number
of servers located directly at the client site. Where noted, client specific credentials may be
required to gain access. The credentials for both the SpryWare Hosted Datacenter and the unique
client site servers is documented in the LetMeIn Spreadsheet found in the Operations folder of the
SpryWare SharePoint site. This documented is password protected using the SpryWare Operations
standard password.
SOP - Master Page 67
14. COMMUNICATION & TRAINING
14.1 REPORTING
14.1.1 Build Lists
14.1.2 Release Notes
14.1.3 Defect/Bug Reports
14.1.4 Status Reports
14.2 STATUS MEETINGS
Staff
Development
Operations & Support
Sales
Client
14.3 TRAINING DOCUMENTATION
14.3.1 KnowledgeQuest
This is a program from another company I worked for. Basically, one set day per week we would
have a lunch meeting. Employees typically brought their own lunch but periodically the company
would pay for it. During the meeting one team member would talk about a topic related to their
job. Perhaps Ron provides a QA overview one day, Terry talks about What’s Up, Claus provides a
Marketing overview etc. The theory is the more we know about what everyone else is doing the
more efficient the company becomes.
SOP - Master Page 68
15. TECHNICAL OPERATIONS
15.1 OVERVIEW
15.2 SYSTEM MONITORING
15.2.1 Server Availability & Performance
15.2.2 Application Software
15.2.3 Website
15.3 PROBLEM MANAGEMENT
15.3.1 Overview
Problems are reported in a number of ways. Clients may call in or send an email to
[email protected]. In either case a CRM Case should be created and used to track the
incident to resolution. System Monitoring though applications like Nagios are used to identify
problem areas on the servers allowing for a proactive approach to addressing the issue or
immediately identifying a problem affecting functionality.
SpryWare addresses problems in real time during normal business hours. For after hours support,
the SpryWare Operations & Support teams follows the “best efforts” support model as outlined in
the Service Level Agreement and Customer Service Guide sections of this document.
For client impacting issues, ongoing communications with the client are critical. These methods of
communication will include phone calls, emails and when requested a Root Cause Analysis (RCA)
report.
SOP - Master Page 69
15.4 PROBLEM DESCRIPTION & RESOLUTION
The remainder of this section will document typical problem conditions and how to address them.
15.4.1 ALERT !!! mail.spryware.local ques is x items (ADM1)
Nearly all of the email that comes out of Chicago Equinix goes through the smtp mail server on
ADM1. This is “mail.spryware.local”.
This includes all Nagios alarms and individual MIS notification emails. An issue has been witnessed
in the past where a device has sent out a large number of emails, or sending malformed messages,
thus preventing further emails from leaving ADM1.
A script has been written to check for this situation.
\\adm1\MisVerify\checkmailqueue.cmd
This script will generate an email which will go out (won’t be stuck) alerting the Operations group of
the condition. The title of this section represents the subject line of the email sent out where “x”
represents the number of stuck emails. The body of the email will provide a listing of “To:”, “From:”,
and “Subject”. The script is run every 15 minutes via Scheduled Tasks.
Upon receiving this ALERT email, a member of the Operations/Support team should proceed as
follows to address the situation:
1. Reply All to the ALERT email stating that you’re going to address the issue.
2. Find the most likely culprit emails listed in the ALERT email. (PINKSHEETS has been the
instigator in the past so will be the example here).
3. RDP into ADM1 and proceed to the C:\MisVerify\ directory.
4. Run the script: killmail.cmd PINKSHEETS (Note, the parameter that follows killmail.cmd can
contain any string listed in the Subject
5. Change directory to mail queue – cd \Intetpub\mailroot\Queue\
6. Check that messages are now being sent out (number should go down after a few minutes)
7. Send a follow up email to Operations stating that the issue has been resolved.
SOP - Master Page 70
15.5 CHANGE MANAGEMENT
15.5.1 Overview
The SpryWare environments are dynamic with frequent updates occurring throughout the week.
These changes consist of but aren’t limited to, SpryWare software upgrades, MIS.XML changes,
hardware upgrades, Operating System patching and network changes.
Change requests can be divided into one of three categories; Customer Requests, Internal Changes
and Exceptions. This section describes each of these categories and details the SpryWare Policies &
Procedures for handling these requests.
15.5.2 Customer Request
Customer requests consist of enhancements, additions or deletions of existing service. These
requests are typically passed to the SpryWare Support team directly by the client or possibly from
the client’s SpryWare Account Manager (Salesman). The procedures for processing these requests
are listed as follows:
Through discussion with the client and/or SpryWare Account Manager make sure the
requirements are fully understood.
Document the Change Request in a CRM Work Order. The work order should contain all
essential information required to complete the request.
15.5.3 Internal Changes
Internal changes relate to SpryWare’s internal infrastructure (Cloud, Demo, NOC, NAS, etc.), or
address changes to the client environment as identified by a SpryWare team member. Examples of
these types of changes include, Operating System Upgrade, SpryWare software upgrade, addition of
memory and network changes. The procedures for processing Internal Changes are similar to those
of Customer Requests:
Document the change in a CRM Work Order.
15.5.4 Policy & Procedures
15.5.4.1 Equipment Purchasing
15.5.4.2 Hardware Upgrades/Replacement
15.5.4.3 Scheduling Downtime
15.5.4.4 Client Site Upgrade Procedures
SOP - Master Page 71
15.6 PROCESS MANAGEMENT
15.6.1 Holiday Procedures
As stated in the Hours of Operation section of this document, the SpryWare office follows the
Holiday Schedule set by the NYSE regarding the days (or partial days) the office will be closed.
Regardless of the status of the office, care must be taken to ensure that all SpryWare servers are
configured properly to handle changes to the various Exchange feed schedules related to holidays.
The scheduling of up & down times associated with each Exchange feed are configured on each
DMS. These schedules do not take into account holiday schedules. Consequently, special
procedures need to be implemented to ensure that these Exchange feeds affected by the holiday
schedule remain offline during the holiday period and come up properly afterword.
Holidays may be global in nature, affecting all exchanges – New Years Day as an example. Holidays
also may be unique to one country. For example, Canada and the LSE won’t recognize the U.S.’
Independence Day. The general procedures associated with the handling of each holiday are the
same. The differences related to each holiday will be related to which lines are affected and the
associated timing.
At the weekly Operations meeting held immediatelyprior to the holiday, the upcoming holiday
procedures are reviewed. The Manager of Operations posts an ALL.PRO file which instructs all
SpryWare DMS boxes to manual down specific lines and later restore the affected lines to the
standard schedule once the holiday is over.
On a case by case basis, some SpryWare clients have requested emails notifying them when the
lines are brought down and then backup again. This client list is reviewed at the Operations
meeting.
In addition to the posting of the ALL.PRO file, the HDS servers, (currently MIS74 & NJMIS21 as of this
writing on November 23, 2012) need to have the “early close” schedule set.
15.6.2 Upgrade Candidate Identification
15.6.3 Client Communications
15.6.4 Application of Patches
15.6.5 Prine
15.6.6 ADM1-3
Include MIS01, ESXi and other support systems
15.6.7 NAS
15.7 INVENTORY CONTROL
15.8 CONTINGENCY PLANNING
15.9 ISSUE TRACKING
SOP - Master Page 72
15.9.1 Defect – Functionally Correct, Inelegant Implementation
15.9.2 Bug – Functionally Incorrect or Non-Functional
SOP - Master Page 73
16. CUSTOMER SUPPORT
CRM as the communication tool
Mention SLA
On-site meetings
Regular conference calls with clients
monitoring
SOP - Master Page 74
17. QUALITY ASSURANCE
17.1 SCOPE DEFINITION
17.2 OK TO DEVELOP
17.3 UNIT TESTING
17.4 PUSH TO QA
17.5 QUALITY CONTROL TESTING
17.6 PUSH TO BETA (CONTROLLED PRODUCTION RELEASE)
17.7 USER ACCEPTANCE
17.8 PUSH TO PRODUCTION
17.9 ISSUE REPORTING/RESOLUTION
SOP - Master Page 75
18. DEVELOPMENT
SOP - Master Page 76
19. SALES & MARKETING
SOP - Master Page 77
20. GENERAL OFFICE
20.1 NEW EMPLOYEE
It’s important to make a new employees first day at SpryWare as enjoyable and useful as possible.
Perform a series of tasks as outlined below to ensure the environment is ready. Show SpryWare as
a together organization.
Following an employee accepting an offer from SpryWare the following tasks need to be completed
prior to their first day in the office. The tasks listed below focus on an employee that will be
working out to of the Chicago Corproate HQ location. Procedures for an employee based out of
New York will be highlighted as will procedurs for employees based at another location.
Cubicle
IT Credentials
o Domain Account
o Intermedia
o TriVenture
o IM
Computer
o Acquire Computer
o Network Connection
o MS Office
o VPN
o Software – May be installed by employee
Telephone
o Extension
o Voice Mail
o Update SharePoint
o Training
Office Supplies
Business Cards
Benefits Package & Payroll
Office Card
Equinix Access if necessary
SpryWare Orientation including Lunch
SOP - Master Page 78
20.2 TELEPHONE SYSTEM
20.3 LAPTOP/DESKTOP
20.4 PRINTERS
SOP - Master Page 79
20.5 GOTOMEETING
SOP - Master Page 80
21. PREFERRED VENDOR LIST
21.1 VENDOR SUPPORT
Highlight assistance these vendors supply and how best to take advantage of these relationships.
21.1.1.1 Essex Radez
21.1.1.2 Equinix
21.1.1.3 Dell
21.1.1.4 HP
21.1.1.5 Telephone
21.1.1.6 Network
21.1.1.7 NY Office
21.1.1.8 InterNap