Installing Apache Hadoop on Eucalyptus using Hortonworks Data Platform stack and Apache Ambari

This post demonstrates Hadoop deployment on Eucalyptus using Apache Ambari on Horton Data Platform stack. Bits and pieces:

  1. Eucalyptus 4.0.0
  2. Hortonworks Data Platform (HDP) stack
  3. Apache Ambari
  4. 4 instance-store/EBS-backed instances
    1. 2 vcpus, 1024MB memory, 20GB disk space
    2. CentOS 6.5 base image

For this is demo we tried to use very minimum resources. Our Eucalyptus deployment topology looks like below, 1x (Cloud Controller + Walrus + Storage Controller + Cluster Controller) 1x (Node Controller + Object Storage Gateway) To meet our instance requirement, we changed the instance-type according to our need.

AVAILABILITYZONE|- m1.xlarge   0003 / 0004   2   1024    20

Preparation Run an instance of m1.xlarge or any other instance type that meets the above requirement. When the instance is running copy the keypair that is used to run this instance at .ssh/id_rsa, we will be using this same keypair for all the instances.

[root@b-11 ~]# euca-describe-instances
RESERVATION	r-46725ad6	691659499425	default
INSTANCE	i-b65be3ed	emi-a4aac1df	euca-172-27-227-26.eucalyptus.internal	running	sshlogin	0		m1.xlarge	2014-06-25T10:39:37.952Z	PARTI00				monitoring-disabled			instance-store					hvm			sg-57c0750a
TAG	instance	i-b65be3ed	euca:node

[root@b-11 ~]# scp -i sshlogin sshlogin

Run three more instance of same type with same keypair and image and copy their private IPs for later use. Add security group rules for Apache Ambari,

[root@b-11 ~]# euca-authorize -P tcp -p 8080 default
[root@b-11 ~]# euca-authorize -P tcp -p 0-65535 -s default

Set Apache Ambari and HDP repository inside the instance,

P. S. [root@euca-172-27-227-26 ~] refers to the Eucalyptus instance

[root@euca-172-27-227-26 ~]# wget -O /etc/yum.repos.d/ambari.repo

[root@euca-172-27-227-26 ~]# wget -O /etc/yum.repos.d/hdp.repo

Install Apache Ambari,

[root@euca-172-27-227-26 ~]# yum install ambari-server -y

Configure Apache Ambari,

[root@euca-172-27-227-26 ~]# ambari-server setup

[Select all default options, unless necessary] Start the Apache Ambari server,

[root@euca-172-27-227-26 ~]# ambari-server start

Our Apache Ambari installation is completed. Now login to the Ambari dashboard with username ‘admin’ and password ‘admin’ at http://instance-public-ip:8080 Installing Hadoop using HDP stack

  • Add a cluster name
  • From Select Stack menu, select HDP 2.1 stack
  • In Install Options menu, add the private IPs/hostnames and private key that we copied previously at .ssh/id_rsa
Ambari Hadoop Installation
Ambari Hadoop Installation
  • Confirm Hosts view will check the hosts availability status and reports back. All green indicates that we are good to go.
Ambari Hadoop Installation
Ambari Hadoop Installation
  • For this demo select all services from Choose Services view
  • In Assign Masters view Ambari will be suggesting a reasonable settings, for this demo we will be using the defaults
  • Again, in Assign Slaves and Clients view, we will stick with the defaults
Ambari Hadoop Installation
Ambari Hadoop Installation
  • In the Customize Services view set the credentials for Hive, Oozie and Nagios.
  • Review and then Install, Start and Test
Ambari Hadoop Installation
Ambari Hadoop Installation

And there is our Hadoop cluster with Hortonworks HDP stack on top of Eucalyptus. Happy Hadooping!