Apache: Big Data North America 2017

Deploying software to secure, clustered Hadoop environments is a challenge. Particularly, one must distribute keytabs, user identities and cluster configuration to build systems like Jenkins; to speak nothing of network access to clusters. At Bloomberg, we ensure our clusters are defined via configuration management and can be automatically configured, operated. Application (HBase, Spark) deployment is a key part of this as well!

We have extended Oozie to provide deployment mechanisms for Git and plans for Maven artifacts allowing us to specify all cluster configuration including software deployed to that cluster. Often this consists of an Oozie workflow to deploy software, allowing for deployments to happen as the permissioned role account and not as a superuser.

Clay will walk through the process of these deployments and the code necessary to make these first-class Oozie actions.