Apache: Big Data North America 2017

Back in the days, you had a single machine and you could scroll down the single log file to figure out what is going on. In this Big Data world you need to combine a lot of logs together to figure out what is going on. Data is coming in huge volumes, with high speed so choosing important information and getting rid of noise becomes real challenge. There is a need for a centralized monitoring platform which will aid the engineers operating the systems, and serve the right information at the right time. This talk will focus on monitoring stack we like to use including Riemann, InfluxDB, ELK and Grafana. Cassandra will be used as an example of distributed system. Problem will be separated in two domains: metric collection and log collection and we will finish with example how you can combine both to pinpoint issues.