Apache: Big Data North America 2017 will be held at the Intercontinental Miami in Miami, Florida. 

Register now for the event taking place May 16-18, 2017. 
Back To Schedule
Wednesday, May 17 • 10:15am - 11:05am
Evolution of an Apache Spark Architecture for Processing Game Data - Nick Afshartous, Warner Brothers Interactive Entertainment (WBIE)

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
We discuss lessons learned from our first production deployment of a Spark Streaming pipeline for processing game data. Deployment is to the AWS Cloud where we use managed services (i.e. EMR, S3 and Redshift). However, having downstream dependencies with outages and unpredictable response latencies can pose significant challenges. To address, we evolved the architecture by separating data processing from post-processing tasks (i.e. copying data into Redshift). Post-processing tasks are sent downstream from Spark to a task executor that was built using Akka Streams and Reactive Kafka. The end result is a loosely coupled architecture where the Spark streaming job is a firehose to S3 and fault-tolerant when Redshift is unavailable.

avatar for Nick Afshartous

Nick Afshartous

Tech Director, Warner Brothers Interactive
Nick Afshartous is a Tech Director at Warner Brothers Interactive Entertainment (WBIE) where he leads the Analytics Core Platform team.   Using Apache Spark, he's helping to build WBIE's next generation real-time analytics platform for processing game data. He's passionate about... Read More →

Wednesday May 17, 2017 10:15am - 11:05am EDT
  Use Cases
  • Experience Level Any