Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
Apache: Big Data North America 2017 will be held at the Intercontinental Miami in Miami, Florida. 

Register now for the event taking place May 16-18, 2017. 
View analytic
Wednesday, May 17 • 4:40pm - 5:30pm
Automation of Rolling Upgrade for Hadoop Cluster without Data Lost and Job Failures - Hiroyuki Adachi & Hiroshi Yamaguchi, Yahoo Japan Corporation

Sign up or log in to save this to your schedule and see who's attending!

Feedback form is now closed.
We present how we automated rolling upgrade for our production Hadoop cluster without data lost and job failures. Apache Ambari can perform rolling upgrade, however it does not consider data lost and effects for running jobs. Therefore, we decided to customize it for our environment and created upgrade procedures with more secure checking. First, we made a custom service for Ambari which operates some functions such as NameNode F/O and load balancer In/Out. Second, we used Ansible which is a configuration management tool to control upgrading task. It automates calling Ambari APIs including the custom service functions, checking cluster statuses (e.g., missing blocks), and running service check jobs while upgrading each component. Consequently, we achieved the automatic rolling upgrade, and we reduced operating costs and minimized inconvenience to users.

Speakers
avatar for Hiroyuki Adachi

Hiroyuki Adachi

Yahoo Japan Corporaion
Hiroyuki Adachi is in charge of DevOps at Hadoop of Yahoo! JAPAN.
avatar for Hiroshi Yamaguchi

Hiroshi Yamaguchi

Yahoo Japan Corporation
Hiroshi Yamaguchi is in charge of DevOps at Hadoop of Yahoo! JAPAN.


Wednesday May 17, 2017 4:40pm - 5:30pm
Windsor

Attendees (16)