Run Appscale on Eucalyptus

Cloud computing is the use of computing resources (hardware and software) that are delivered as a service over a network (typically the Internet). – Wikipedia

According to Wikipedia currently there are few popular service models exist.

1. Infrastructure as a service (IaaS)
2. Platform as a service (PaaS)
3. Software as a service (SaaS)

So, I have an Eucalyptus cloud, which is great, serves as AWS-like IaaS platform. But now I want PaaS. And right here Appscale comes into play with full compatibility of Google App Engine (GAE) applications. In this post, we will install the popular open source PaaS framework Appscale on Eucalyptus, the AWS compatible open source IaaS platform.

Agenda
0. Introduction
1. Resize Lucid image
2. Install Appscale from source
3. Install Appscale Tool
4. Bundle Appscale image
5. Run Appscale
6. Run an application on Appscale

Eucalyptus
Eucalyptus Cloud platform is open source software for building AWS-compatible private and hybrid clouds. It pools together existing virtualized infrastructure to create cloud resources for compute, network and storage.

Appscale
AppScale is an open-source framework for running Google App Engine applications. It is an implementation of a cloud computing platform (Platform-as-a-Service), supporting Xen, KVM, Amazon EC2 and Eucalyptus. It has been developed and is maintained by AppScale Systems.

Resize Lucid image

Download the Lucid image,
wget http://cloud-images.ubuntu.com/lucid/current/lucid-server-cloudimg-amd64.tar.gz

To install Appscale we need a bigger space than it comes by default with the image. Let’s increase the root size of the image,

dd if=/dev/zero of=lucid-server20g.img bs=1M count=20480

mkfs.ext3 -F lucid-server20g.img

mkdir new
mkdir old

mount -o loop lucid-server-cloudimg-amd64.img old/
mount -o loop lucid-server20g.img new/

(cd old; tar czf - .) | (cd new; tar xzf -)

umount old
umount new

[detail from Eucalyptus knowledge base article]

So, the problem I was facing, the modified image wasn’t working properly with the kernel/ramdisk those came with tar file. My colleague Deependra figured out the problem gets resolved with the latest kernel. So, you can either start up an instance upgrade with the 3.0.0-31 kernel or download the updated kernel and ramdisk from the following link as long as they are there,

http://dl.dropbox.com/u/5458574/initrd.img-3.0.0-31-server
http://dl.dropbox.com/u/5458574/vmlinuz-3.0.0-31-server

Now follow Eucalyptus Administration Guide to add the image.

Run an instance to build the Appscale image,

euca-run-instances -k sshlogin -t m1.large emi-7A453BC1

Install Appscale from source

apt-get install -y git-core
git clone git://github.com/AppScale/appscale.git --branch master
cd appscale/debian
bash appscale_build.sh

Install Appscale Tool

git clone git://github.com/AppScale/appscale-tools.git --branch master
cd appscale-tools/debian
bash appscale_build.sh

export PATH=$PATH:/usr/local/appscale-tools/bin

add-apt-repository "deb http://downloads.eucalyptus.com/software/euca2ools/2.1/ubuntu lucid main"

Bundle Appscale image

Now bundle the instance and register it,

euca-bundle-vol -p appscale-1-6-7-8g -d ./ -e /var/lib/dhcp3 -s 8192
euca-upload-bundle -b demo -m ./appscale-1-6-7-8g.manifest.xml
euca-register demo/appscale-1-6-7-8g.manifest.xml

Run Appscale

euca-run-instances -k sshlogin -t m1.large emi-D4A33E62

Now login to the Appscale instance. Run the following to create the configuration file,

appscale init cloud

Now modify the AppScalefile file to be used by Appscale.

So for this demo, our minimal configuration file looks like following,

infrastructure : 'euca'
machine : 'emi-D4A33E62'
instance_type : 'c1.xlarge'
table : 'cassandra'
keyname : 'appscalekey'
group : 'appscalegroup'
verbose : True
min : 3
max : 3

The machine would be the emi which has Appscale installed on it, instance_type we prefer to have the biggest possible one, for this demo we will be using default table, we turned on the verbose flag on, since this is a test environment, we also interested in the power of cloud service, so we set min, max to 3, this will spawn three instances and replicate data base 3 times.

So, now we are all set to start the Appscale platform. Run the following to start up Appscale,

appscale up

So after all those verbose output, now we see the following,

AppScale successfully started!
View status information about your AppScale deployment at http://10.104.3.129/status


And if we check http://10.104.3.129/status, we should see the healthy status

Appscale API status

Run an application on Appscale

We will be running a simple chat application that is available on the google app engine samples. So we save the following app and unzip it,
https://google-app-engine-samples.googlecode.com/files/simpleajaxchat_10312008.zip

Run the following command to deploy the first application,

appscale deploy ~/samples/simpleajaxchat/

And the happy line would look like the following,
Your app can be reached at the following URL: http://10.104.3.129:8080

Sample chat application

And now we have our first application hosted on Appscale PaaS platform.

For more monitoring, login the the Appscale console and select monitoring data.

Appstale status from command line,

appscale status

For troubleshooting Appscale comes with a nice tool. Run the following to gather all the logs at one place,

appscale logs /path/filename

Login to the master node,

appscale ssh

Shutdown Appscale,

appscale down

There is a lot more available on github,
https://github.com/AppScale/

Also you can submit your issues here,
https://github.com/AppScale/appscale/issues

Try out the latest Eucalyptus code from github,
https://github.com/eucalyptus/eucalyptus

We are always open to your comments and feedback,
https://eucalyptus.atlassian.net/
https://engage.eucalyptus.com/

Eucalyptus HA and lock files

Eucalyptus comes with its own High Availability (HA) feature from version 3.0 and with the passage of time, it is becoming more and more stable with every release.

With recent release (3.2.0 and onwards), it has got few behavioral changes, which of course makes it more robust. To describe it in one sentence, “Lock files are to prevent data loss during HA failover”. So, lock files are to ensure that Cloud Controller (CLC) database synchronization happens properly and in any case it does not end-up in split-brain situation.

In the recent versions, Eucalyptus introduces two kinds of lock files, disabled.lock and partitioned.lock.

disabled.lock
This file is generated to enforce the correct order of starting CLC services. So, if you stop the disabled CLC disabled.lock file is generated, then you stop the enabled CLC and try to start the disabled CLC, it won’t start, unless the cloud administrator enforce by deleting the disabled.lock file manually.

partitioned.lock
On the other hand this file is generated basically to protect split brain situation. It is to prevent two CLCs to go into enabled state. When ever there is a network outage between CLC <-> CLC, the partitioned.lock file is generated. If we see more closely, when the database becomes inaccessible, the partitioned.lock file is generated.

When the CLC fails to start because of disabled.lock or partitioned.lock Eucalyptus populates logs in cloud-fault.log. This basically represents that Eucalyptus cannot proceed by itself any further and it requires manual involvement of cloud administrator.

These are the basic use cases of both lock files. partitioned.lock is little more aggressive than disabled.lock. It actually has few more use cases, which I am yet to figure out though. In brief, when the database is killed on the enabled CLC a partitioned.lock can be generated or when the public interface is down on the enabled CLC it generates partitioned.lock file, it happens even if it’s a non-HA setup. Another case would be, if the enabled CLC cannot talk to it’s database and it finds another database on a different host Eucalyptus generates partitioned.lock.

This may not be all, the code is being improved everyday, but this is the last known behavior of the lock files.

Comments and suggestions are most welcome.

Eucalyptus and Nagios

Reblogged from nurmiblog:

Production deployments of Eucalyptus, like production deployments of any infrastructure software running in a data center, require some amount of health and status monitoring be happening in order to both allow the Eucalyptus/data-center administrator the ability to stay on top of evolving resource situations and to provide invaluable diagnostic information when something is going sideways within the resource pool.  Fortunately for all of us, there exists a wide variety of health/status monitoring system out there, and several of them are of extremely high quality, tried and tested, and are available as part of major Linux distributions as pre-packaged open-source solutions.  

Read more… 1,429 more words

Contributing to Eucalyptus made easy

The power of open source is incredible. Don’t believe me? Click here or here Contribution is something that makes open source projects most powerful. It has been proven with Linux a long time ago. Also, giant companies have started to believe in open source. Open source is not just about making a product free to use, what I believe is, it’s about giving users the freedom to do whatever they want to do with a product, with the source code.

But sometimes, I feel, is it always that easy to contribute? Well, that’s certainly a question. Being an user, it often needs some guidance to get started with something. Why do I believe that? Well, you will know, if you are in such a place, where you are the only person doing something alone

Eucalyptus comes with such a license, that allows its users to modify the codes as they want and if someone wants to contribute, s/he is always welcome to do so.

Detail on Contributing Code to Eucalyptus

So, how do we start?

Here is the few thing that we will be using,
1. A machines, with CentOS 6.3 installed
2. src.rpm of Eucalyptus 3.1.2

Set up build dependency repository,

cat <<EOF >/etc/yum.repos.d/builddeps.repo
[builddeps]
name=builddeps
baseurl=http://downloads.eucalyptus.com/software/eucalyptus/build-deps/3.1/centos/6/x86_64/
gpgcheck=0
enabled=1
EOF

Configure the EPEL package repository,

yum install http://downloads.eucalyptus.com/software/eucalyptus/3.1/centos/6/x86_64/epel-release-6.noarch.rpm -y 

Download the source RPM,

wget -nd http://downloads.eucalyptus.com/software/eucalyptus/3.1/centos/6/source/eucalyptus-3.1.2-0.210.el6.src.rpm

Download and install build-dependencies,

yum install -y rpm-build gcc make

Install the build dependencies for the source RPM,

yum-builddep --nogpgcheck -y eucalyptus-3.1.2-0.210.el6.src.rpm

Install the source RPM,

rpm -ivh eucalyptus-3.1.2-0.210.el6.src.rpm

At this point we have eucalyptus.spec file ready. Run the following commands to get the bits in the ~/rpmbuild/BUILD directory.

cd ~/rpmbuild/SPECS
rpmbuild -bp eucalyptus.spec

Make a patch from the original bits,

cp -r eucalyptus-3.1.2 eucalyptus-3.1.2.orig
vim eucalyptus-3.1.2/<file_that_needs_aid>
diff -Npru eucalyptus-3.1.2.orig eucalyptus-3.1.2 > <name_the_patch>.patch
cp <name_the_patch>.patch ~/rpmbuild/SOURCES/

Now, modify the spec file from the ~/rpmbuild/SPECS/ and add the patch into it.

Build SRPMs and RPMs,

rpmbuild -ba eucalyptus.spec

Test your newly created RPMs and contribute to the community.

Happy patching!

Contribution is not only limited to source code.

File bug reports: eucalyptus.atlassian.net

Send a pull request on Github: https://github.com/eucalyptus/

Also, tell us about your problem with Eucalyptus on Engage: engage.eucalyptus.com

Eucalyptus Source Installation

Eucalyptus 3.2 is becoming ready. It is going to be out soon. But if you think, it’s too hard for you to wait till GA, there are two solutions for you to try out. You can check out nightly builds or go for source installation, which is *fairly* easy.

For this installation I am using two CentOS 6.3 boxes, one as Frontend (CLC/Walrus/CC/SC) and another for backend (NC).

Download the source code from Github,

 # git clone git://github.com/eucalyptus/eucalyptus --recursive 

Configure repositories,

 # yum --nogpg install http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-7.noarch.rpm -y 

[walrus host only]

 # yum --nogpg install http://elrepo.org/linux/elrepo/el6/x86_64/RPMS/elrepo-release-6-4.el6.elrepo.noarch.rpm -y 

Create a eucadeps.repo file for eucalyptus dependencies,

[euca-deps]name=euca-deps
baseurl= http://downloads.eucalyptus.com/software/eucalyptus/build-deps/3.1/rhel/$releasever/$basearch
gpgcheck=0
enabled=1

[euca-runtime]
name=euca-deps
baseurl= http://downloads.eucalyptus.com/software/eucalyptus/runtime-deps/3.1/rhel/$releasever/$basearch
gpgcheck=0
enabled=1

[euca2ools]
name=euca2ools
baseurl= http://downloads.eucalyptus.com/software/euca2ools/2.1/centos/$releasever/$basearch
gpgcheck=0
enabled=1

Install the required dependencies,

 # yum install gcc make ant ant-nodeps axis2-adb-codegen axis2-codegen axis2c-devel jpackage-utils libvirt-devel libxml2-devel libxslt-devel python-devel python-setuptools rampartc-devel swig velocity xalan-j2-xsltc gawk java-devel java-devel openssl-devel curl-devel axis2c dhcp41 dhcp41-common postgresql91 postgresql91-server python-boto rampartc iscsi-initiator-utils perl-Crypt-OpenSSL-RSA perl-Crypt-OpenSSL-Random sudo which jpackage-utils java lvm2 velocity drbd83 drbd83-kmod drbd83-utils drbd euca2ools bitstream-vera-fonts dejavu-serif-fonts bridge-utils iptables vtun device-mapper coreutils e2fsprogs file parted util-linux rsync PyGreSQL libcurl curl httpd libvirt scsi-target-utils vconfig wget kvm patch 

Download euca-WSDL2C.sh and save it to /opt on both machines,

 # wget https://raw.github.com/eucalyptus/eucalyptus-rpmspec/master/euca-WSDL2C.sh -O /opt/euca-WSDL2C.sh 

Download and save eucalyptus-nc-libvirt.pkla to /var/lib/polkit-1/localauthority/10-vendor.d/

 # wget https://raw.github.com/eucalyptus/eucalyptus/master/tools/eucalyptus-nc-libvirt.pkla -O /var/lib/polkit-1/localauthority/10-vendor.d/eucalyptus-nc-libvirt.pkla 

Add user “eucalyptus” on frontend,

 # adduser eucalyptus 

On NC,

 # useradd -G kvm eucalyptus 

Lets build Eucalyptus source now.

Set the following paths on both machines,

# export EUCALYPTUS="/"

Now start compiling Eucalyptus on both machines. Go to the eucalyptus directory and run the following,

 # ./configure '--with-axis2=/usr/share/axis2-*' --with-axis2c=/usr/lib64/axis2c --prefix=$EUCALYPTUS --with-apache2-module-dir=/usr/lib64/httpd/modules --with-db-home=/usr/pgsql-9.1 --with-wsdl2c-sh=/opt/euca-WSDL2C.sh 

 # make clean ; make ; make install 

Add the following in /etc/sysconfig/network,

 NOZEROCONF=yes 

On NC run the following,

# modprobe kvm_intel
# modprobe vhost_net

Configure eucalyptus.conf on both machines.

Run the following on both machines to set permission properly,

 # su -c "euca_conf --setup" 

On Frontend, initialize database and start eucalyptus-cloud service, which will start Cloud Controller, Walrus, Storage Controller, as we have put all in one box. And then start the Cluster Controller. Finally, start Node controller.

# euca_conf --initialize
# service eucalyptus-cloud start

# service eucalyptus-cc start

# service eucalyptus-nc start

Register Eucalyptus components,

# euca_conf --register-walrus --partition walrus --host 10.104.1.179 --component W00
# euca_conf --register-cluster --partition cluster01 --host 10.104.1.179 --component CC00
# euca_conf --register-sc --partition cluster01 --host 10.104.1.179 --component SC00

From 3.2.0, it is a must to set storage manager,

 # euca-modify-property -p cluster01.storage.blockstoragemanager=overlay 

Register node,

 # euca_conf --register-nodes 10.105.1.180 

Good Luck!