Fantasy Football FisherPaul Singman

Insight Data Engineering FellowOctober 2016

Motivation

● Current offerings:○ Season long○ Daily

● What’s missing?○ Micro-leagues

1 Day Full SeasonSub 1 minute

Data Simulation and Ingestion

• Play-by-play files obtained from SportRadar API

• Plays simulated at rate of 300 /s

{"player_name": "Marcus Mariota", "timestamp": "2016-09-29_02:19:23", "touchdown": 0, "yards": 9, "position": "QB"}

Example Play

Fantasy Football Fisher Architecture

Instances Cost Total

2 x 4 m4.large

$0.12 /hr $.96 /hr

Use of Windowed Streaming

Use of Windowed Streaming

30 sec 30 sec

Why Windowed Streaming?

Why Windowed Streaming?

○ Receiver creates a blocks of data every Block Interval (200ms)

■ No. of tasks = Batch Interval / Block Interval

○ Ideally No. of tasks = (No. of cores in cluster - 1) * 2 - to - 3

■ 3 cores means 6-9 tasks of data per batch■ Batch Interval of 1.5s is ideal

• Bachelor of Science in Stats from Penn● Shelf full of O’Reilly books

• Jr Data Engineer experience at early-stage startup (Mighty)

• Enjoy movies, backgammon, and rooftop yoga