Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
Apache: Big Data North America 2017 will be held at the Intercontinental Miami in Miami, Florida. 

Register now for the event taking place May 16-18, 2017. 
View analytic
Wednesday, May 17 • 10:15am - 11:05am
Evolution of an Apache Spark Architecture for Processing Game Data - Nick Afshartous, Warner Brothers Interactive Entertainment (WBIE)

Sign up or log in to save this to your schedule and see who's attending!

Feedback form is now closed.
We discuss lessons learned from our first production deployment of a Spark Streaming pipeline for processing game data. Deployment is to the AWS Cloud where we use managed services (i.e. EMR, S3 and Redshift). However, having downstream dependencies with outages and unpredictable response latencies can pose significant challenges. To address, we evolved the architecture by separating data processing from post-processing tasks (i.e. copying data into Redshift). Post-processing tasks are sent downstream from Spark to a task executor that was built using Akka Streams and Reactive Kafka. The end result is a loosely coupled architecture where the Spark streaming job is a firehose to S3 and fault-tolerant when Redshift is unavailable.

Speakers
avatar for Nick Afshartous

Nick Afshartous

Tech Director, Warner Brothers Interactive
Nick Afshartous is a Tech Director at Warner Brothers Interactive Entertainment (WBIE) where he leads the Analytics Core Platform team.   Using Apache Spark, he's helping to build WBIE's next generation real-time analytics platform for processing game data. He's passionate abou... Read More →


Wednesday May 17, 2017 10:15am - 11:05am
Trianon
  • Experience Level Any

Attendees (19)