We present how we automated rolling upgrade for our production Hadoop cluster without data lost and job failures. Apache Ambari can perform rolling upgrade, however it does not consider data lost and effects for running jobs. Therefore, we decided to customize it for our environment and created upgrade procedures with more secure checking. First, we made a custom service for Ambari which operates some functions such as NameNode F/O and load balancer In/Out. Second, we used Ansible which is a configuration management tool to control upgrading task. It automates calling Ambari APIs including the custom service functions, checking cluster statuses (e.g., missing blocks), and running service check jobs while upgrading each component. Consequently, we achieved the automatic rolling upgrade, and we reduced operating costs and minimized inconvenience to users.
