Run Appscale on Eucalyptus

Cloud computing is the use of computing resources (hardware and software) that are delivered as a service over a network (typically the Internet). – Wikipedia

According to Wikipedia currently there are few popular service models exist.

1. Infrastructure as a service (IaaS)
2. Platform as a service (PaaS)
3. Software as a service (SaaS)

So, I have an Eucalyptus cloud, which is great, serves as AWS-like IaaS platform. But now I want PaaS. And right here Appscale comes into play with full compatibility of Google App Engine (GAE) applications. In this post, we will install the popular open source PaaS framework Appscale on Eucalyptus, the AWS compatible open source IaaS platform.

Agenda
0. Introduction
1. Resize Lucid image
2. Install Appscale from source
3. Install Appscale Tool
4. Bundle Appscale image
5. Run Appscale
6. Run an application on Appscale

Eucalyptus
Eucalyptus Cloud platform is open source software for building AWS-compatible private and hybrid clouds. It pools together existing virtualized infrastructure to create cloud resources for compute, network and storage.

Appscale
AppScale is an open-source framework for running Google App Engine applications. It is an implementation of a cloud computing platform (Platform-as-a-Service), supporting Xen, KVM, Amazon EC2 and Eucalyptus. It has been developed and is maintained by AppScale Systems.

Resize Lucid image

Download the Lucid image,
wget http://cloud-images.ubuntu.com/lucid/current/lucid-server-cloudimg-amd64.tar.gz

To install Appscale we need a bigger space than it comes by default with the image. Let’s increase the root size of the image,

dd if=/dev/zero of=lucid-server20g.img bs=1M count=20480

mkfs.ext3 -F lucid-server20g.img

mkdir new
mkdir old

mount -o loop lucid-server-cloudimg-amd64.img old/
mount -o loop lucid-server20g.img new/

(cd old; tar czf - .) | (cd new; tar xzf -)

umount old
umount new

[detail from Eucalyptus knowledge base article]

So, the problem I was facing, the modified image wasn’t working properly with the kernel/ramdisk those came with tar file. My colleague Deependra figured out the problem gets resolved with the latest kernel. So, you can either start up an instance upgrade with the 3.0.0-31 kernel or download the updated kernel and ramdisk from the following link as long as they are there,

http://dl.dropbox.com/u/5458574/initrd.img-3.0.0-31-server
http://dl.dropbox.com/u/5458574/vmlinuz-3.0.0-31-server

Now follow Eucalyptus Administration Guide to add the image.

Run an instance to build the Appscale image,

euca-run-instances -k sshlogin -t m1.large emi-7A453BC1

Install Appscale from source

apt-get install -y git-core
git clone git://github.com/AppScale/appscale.git --branch master
cd appscale/debian
bash appscale_build.sh

Install Appscale Tool

git clone git://github.com/AppScale/appscale-tools.git --branch master
cd appscale-tools/debian
bash appscale_build.sh

export PATH=$PATH:/usr/local/appscale-tools/bin

add-apt-repository "deb http://downloads.eucalyptus.com/software/euca2ools/2.1/ubuntu lucid main"

Bundle Appscale image

Now bundle the instance and register it,

euca-bundle-vol -p appscale-1-6-7-8g -d ./ -e /var/lib/dhcp3 -s 8192
euca-upload-bundle -b demo -m ./appscale-1-6-7-8g.manifest.xml
euca-register demo/appscale-1-6-7-8g.manifest.xml

Run Appscale

euca-run-instances -k sshlogin -t m1.large emi-D4A33E62

Now login to the Appscale instance. Run the following to create the configuration file,

appscale init cloud

Now modify the AppScalefile file to be used by Appscale.

So for this demo, our minimal configuration file looks like following,

infrastructure : 'euca'
machine : 'emi-D4A33E62'
instance_type : 'c1.xlarge'
table : 'cassandra'
keyname : 'appscalekey'
group : 'appscalegroup'
verbose : True
min : 3
max : 3

The machine would be the emi which has Appscale installed on it, instance_type we prefer to have the biggest possible one, for this demo we will be using default table, we turned on the verbose flag on, since this is a test environment, we also interested in the power of cloud service, so we set min, max to 3, this will spawn three instances and replicate data base 3 times.

So, now we are all set to start the Appscale platform. Run the following to start up Appscale,

appscale up

So after all those verbose output, now we see the following,

AppScale successfully started!
View status information about your AppScale deployment at http://10.104.3.129/status


And if we check http://10.104.3.129/status, we should see the healthy status

Appscale API status

Run an application on Appscale

We will be running a simple chat application that is available on the google app engine samples. So we save the following app and unzip it,
https://google-app-engine-samples.googlecode.com/files/simpleajaxchat_10312008.zip

Run the following command to deploy the first application,

appscale deploy ~/samples/simpleajaxchat/

And the happy line would look like the following,
Your app can be reached at the following URL: http://10.104.3.129:8080

Sample chat application

And now we have our first application hosted on Appscale PaaS platform.

For more monitoring, login the the Appscale console and select monitoring data.

Appstale status from command line,

appscale status

For troubleshooting Appscale comes with a nice tool. Run the following to gather all the logs at one place,

appscale logs /path/filename

Login to the master node,

appscale ssh

Shutdown Appscale,

appscale down

There is a lot more available on github,
https://github.com/AppScale/

Also you can submit your issues here,
https://github.com/AppScale/appscale/issues

Try out the latest Eucalyptus code from github,
https://github.com/eucalyptus/eucalyptus

We are always open to your comments and feedback,
https://eucalyptus.atlassian.net/
https://engage.eucalyptus.com/

Eucalyptus HA and lock files

Eucalyptus comes with its own High Availability (HA) feature from version 3.0 and with the passage of time, it is becoming more and more stable with every release.

With recent release (3.2.0 and onwards), it has got few behavioral changes, which of course makes it more robust. To describe it in one sentence, “Lock files are to prevent data loss during HA failover”. So, lock files are to ensure that Cloud Controller (CLC) database synchronization happens properly and in any case it does not end-up in split-brain situation.

In the recent versions, Eucalyptus introduces two kinds of lock files, disabled.lock and partitioned.lock.

disabled.lock
This file is generated to enforce the correct order of starting CLC services. So, if you stop the disabled CLC disabled.lock file is generated, then you stop the enabled CLC and try to start the disabled CLC, it won’t start, unless the cloud administrator enforce by deleting the disabled.lock file manually.

partitioned.lock
On the other hand this file is generated basically to protect split brain situation. It is to prevent two CLCs to go into enabled state. When ever there is a network outage between CLC <-> CLC, the partitioned.lock file is generated. If we see more closely, when the database becomes inaccessible, the partitioned.lock file is generated.

When the CLC fails to start because of disabled.lock or partitioned.lock Eucalyptus populates logs in cloud-fault.log. This basically represents that Eucalyptus cannot proceed by itself any further and it requires manual involvement of cloud administrator.

These are the basic use cases of both lock files. partitioned.lock is little more aggressive than disabled.lock. It actually has few more use cases, which I am yet to figure out though. In brief, when the database is killed on the enabled CLC a partitioned.lock can be generated or when the public interface is down on the enabled CLC it generates partitioned.lock file, it happens even if it’s a non-HA setup. Another case would be, if the enabled CLC cannot talk to it’s database and it finds another database on a different host Eucalyptus generates partitioned.lock.

This may not be all, the code is being improved everyday, but this is the last known behavior of the lock files.

Comments and suggestions are most welcome.

Eucalyptus and Nagios

Reblogged from nurmiblog:

Production deployments of Eucalyptus, like production deployments of any infrastructure software running in a data center, require some amount of health and status monitoring be happening in order to both allow the Eucalyptus/data-center administrator the ability to stay on top of evolving resource situations and to provide invaluable diagnostic information when something is going sideways within the resource pool.  Fortunately for all of us, there exists a wide variety of health/status monitoring system out there, and several of them are of extremely high quality, tried and tested, and are available as part of major Linux distributions as pre-packaged open-source solutions.  

Read more… 1,429 more words

Monitoring with Nagios

With the growing number of servers, monitoring often becomes harder day by day. There are various solutions for this. Two very popular I have seen around are Zenoss and Nagios.

Well, here comes the very common question. Which one to use? The best answer would be, the one with less learning and more features Most often the combination isn’t so easy to achieve.

There are so many comparisons around monitoring tools. I have found these two relevant to this post, one from Linux.com and another from Zenoss.org Help yourself with the links, I am not going to make any comparison between these two.

At Eucalyptus, we have customers who run a lot of servers and monitoring tool is not another fancy product for them, rather it’s a necessity.

But deploying a quick Nagios monitoring system on CentOS is only few steps away. It should not be any different on other distros as well.

Install Nagios packages,

yum install -y nagios nagios-plugins nagios-plugins-nrpe nagios-plugins-all

Set password for Nagios admin,

htpasswd /etc/nagios/passwd nagiosadmin

Start the services and update runlevel for each of them,

chkconfig nagios on
chkconfig httpd on
service nagios start
service httpd start

Define host and services

vim /etc/nagios/objects/host1.cfg
# Define a host

define host{
        use                     linux-server
        host_name               host1
        address                 10.104.3.128
        }

# Define a service

define service{
        use                             generic-service
        host_name                       host1
        service_description             PING
        check_command                   check_ping!100.0,20%!500.0,60%
        }

Add the following line to nagios.cfg,

cfg_file=/etc/nagios/objects/host1.cfg

Reload Nagios service,

service nagios reload

Check Nagios admin panel at http://<ip_address>/nagios

Tada!!

And of course, it is released under the GPL license.

Contributing to Eucalyptus made easy

The power of open source is incredible. Don’t believe me? Click here or here Contribution is something that makes open source projects most powerful. It has been proven with Linux a long time ago. Also, giant companies have started to believe in open source. Open source is not just about making a product free to use, what I believe is, it’s about giving users the freedom to do whatever they want to do with a product, with the source code.

But sometimes, I feel, is it always that easy to contribute? Well, that’s certainly a question. Being an user, it often needs some guidance to get started with something. Why do I believe that? Well, you will know, if you are in such a place, where you are the only person doing something alone

Eucalyptus comes with such a license, that allows its users to modify the codes as they want and if someone wants to contribute, s/he is always welcome to do so.

Detail on Contributing Code to Eucalyptus

So, how do we start?

Here is the few thing that we will be using,
1. A machines, with CentOS 6.3 installed
2. src.rpm of Eucalyptus 3.1.2

Set up build dependency repository,

cat <<EOF >/etc/yum.repos.d/builddeps.repo
[builddeps]
name=builddeps
baseurl=http://downloads.eucalyptus.com/software/eucalyptus/build-deps/3.1/centos/6/x86_64/
gpgcheck=0
enabled=1
EOF

Configure the EPEL package repository,

yum install http://downloads.eucalyptus.com/software/eucalyptus/3.1/centos/6/x86_64/epel-release-6.noarch.rpm -y 

Download the source RPM,

wget -nd http://downloads.eucalyptus.com/software/eucalyptus/3.1/centos/6/source/eucalyptus-3.1.2-0.210.el6.src.rpm

Download and install build-dependencies,

yum install -y rpm-build gcc make

Install the build dependencies for the source RPM,

yum-builddep --nogpgcheck -y eucalyptus-3.1.2-0.210.el6.src.rpm

Install the source RPM,

rpm -ivh eucalyptus-3.1.2-0.210.el6.src.rpm

At this point we have eucalyptus.spec file ready. Run the following commands to get the bits in the ~/rpmbuild/BUILD directory.

cd ~/rpmbuild/SPECS
rpmbuild -bp eucalyptus.spec

Make a patch from the original bits,

cp -r eucalyptus-3.1.2 eucalyptus-3.1.2.orig
vim eucalyptus-3.1.2/<file_that_needs_aid>
diff -Npru eucalyptus-3.1.2.orig eucalyptus-3.1.2 > <name_the_patch>.patch
cp <name_the_patch>.patch ~/rpmbuild/SOURCES/

Now, modify the spec file from the ~/rpmbuild/SPECS/ and add the patch into it.

Build SRPMs and RPMs,

rpmbuild -ba eucalyptus.spec

Test your newly created RPMs and contribute to the community.

Happy patching!

Contribution is not only limited to source code.

File bug reports: eucalyptus.atlassian.net

Send a pull request on Github: https://github.com/eucalyptus/

Also, tell us about your problem with Eucalyptus on Engage: engage.eucalyptus.com