Apache: Big Data North America 2017

At Optimizely we track billions of user events, such as page views, clicks and custom events, on a daily basis to provide our customers with immediate access to key analytics and business insights. Because of this, we are constantly innovating on our data ingestion pipeline. Over the course of development we have moved from batch data ingestion processes, to streaming, to a hybrid or "lambda" approach and back to full streaming again. I will present the technical details and challenges in developing this system, which includes use of Apache Samza, Flume, Kafka, HBase and Hadoop, as well as some of the lessons learned along the way.

This talk will summarize the story we described in this blog post and present where we have gone since: http://highscalability.com/blog/2016/11/16/the-story-of-batching-to-streaming-analytics-at-optimizely.html