Apache: Big Data North America 2017 will be held at the Intercontinental Miami in Miami, Florida. 

Register now for the event taking place May 16-18, 2017. 
Back To Schedule
Wednesday, May 17 • 4:40pm - 5:30pm
Automation of Rolling Upgrade for Hadoop Cluster without Data Lost and Job Failures - Hiroyuki Adachi & Hiroshi Yamaguchi, Yahoo Japan Corporation

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
We present how we automated rolling upgrade for our production Hadoop cluster without data lost and job failures. Apache Ambari can perform rolling upgrade, however it does not consider data lost and effects for running jobs. Therefore, we decided to customize it for our environment and created upgrade procedures with more secure checking. First, we made a custom service for Ambari which operates some functions such as NameNode F/O and load balancer In/Out. Second, we used Ansible which is a configuration management tool to control upgrading task. It automates calling Ambari APIs including the custom service functions, checking cluster statuses (e.g., missing blocks), and running service check jobs while upgrading each component. Consequently, we achieved the automatic rolling upgrade, and we reduced operating costs and minimized inconvenience to users.

avatar for Hiroyuki Adachi

Hiroyuki Adachi

Yahoo Japan Corporaion
Hiroyuki Adachi is in charge of DevOps at Hadoop of Yahoo! JAPAN.
avatar for Hiroshi Yamaguchi

Hiroshi Yamaguchi

Yahoo Japan Corporation
Hiroshi Yamaguchi is in charge of DevOps at Hadoop of Yahoo! JAPAN.

Wednesday May 17, 2017 4:40pm - 5:30pm EDT