Apache: Big Data North America 2017

Hadoop clusters are continuously becoming larger with several thousand machines,running thousands of jobs concurrently on 1000-1500 queues divided by different tenants and crunching higher volume of data than before.Hence,maintaining good performance of such large clusters,ensuring fast recovery times,upgrading them and debugging them becomes a major challenge.With larger clusters,enterprises expect even more efficient cluster utilisation.The fact that jobs are in turn executed as part of a workflow adds to the complexity. As time progresses, clusters would become even larger, i.e. have several tens of thousands of machines.

In this talk, we plan to share issues we came across while handling large clusters and the optimizations we had to make to resolve them.We would also talk about a few upcoming features in Hadoop which aim to overcome challenges posed by clusters at gigantic scale.