Apache: Big Data North America 2017

In this talk, we present design and implementation of MOHA (MTC on Hadoop) framework which can effectively combine Many-Task Computing (MTC) technologies with Big Data platform Hadoop to enable more rich data analytics workflows in the ecosystem. MTC is a new computing paradigm that can consist of, e.g., millions of small tasks where each task communicates through files resulting in another type of data-intensive workload. MOHA is developed as one of YARN applications so that it can transparently cohost existing MTC applications with other Big Data processing frameworks in a single Hadoop cluster. MOHA can substantially reduce the overall execution time of many-task processing with minimal amount of resources compared to an existing Hadoop YARN application by effectively exploiting open-source distributed message queues (Apache ActiveMQ, Kafka) and streamlined task dispatching mechanism.