Eucalyptus VPC with MidoNet 5.2

Eucalyptus started to support AWS compatible VPC (Virtual Private Cloud) from v4.2 as a new networking mode VPCMIDO. Eucalyptus still supports the EC2 classic networking in EDGE networking mode. Eucalyptus VPC exposes the same AWS VPC APIs to support the existing application that were built for AWS. Eucalyptus uses MidoNet as a backend for VPC and supports both open source MidoNet and Midokura Enterprise MidoNet (MEM). The current Eucalyptus release v4.3 supports MidoNet v1.9 and has gone through a huge improvements in terms of performance and stability from Eucalyptus v4.2.2.

Eucalyptus v4.4 is under heavy development and supports current stable release of MidoNet v5.2!

A basic deployment of MidoNet (v5.2) for Eucalyptus VPC consists of following components:

  1. MidoNet Cluster – installed on Cloud Controller (CLC)
  2. Gateway Node (MidoNet Gateway)
  3. Network State Database (NSDB) – Zookeeper and Cassandra
  4. MidoNet Agents (Midolman) – Cloud Controller (CLC) and Node Controllers (NC)

Steps to install Eucalyptus 4.4 VPC

Even though Eucalyptus 4.4 is still under development, nightly packages are already available here.

Installation of the MidoNet components are pretty straight forward and are well explained in MidoNet documentation.

  • Repository configuration for opensource MidoNet
  • Network State Database installation
  • Install and configure MidoNet Cluster on CLC
    # install packages
    yum install midonet-cluster python-midonetclient
    # file: /etc/midonet/midonet.conf
    zookeeper_hosts =
  • Run the following command on MidoNet Cluster, configure access to NSDB
    $ cat << EOF | mn-conf set -t default
    zookeeper {
        zookeeper_hosts = ""
    cassandra {
        servers = ""
  • Start midonet-cluster.service
  • Install and configure Midolman on CLC and NCs
    yum install java-1.8.0-openjdk-headless midolman
    # file: /etc/midolman/midolman.conf
    zookeeper_hosts =
  • Set Midolman resource template
    mn-conf template-set -h local -t default
  • Start midolman.service on all the hosts.
  • Install and configure Eucalyptus with VPCMIDO as networking mode. Eucalyptus 4.4 installation is identical to v4.3.

Create MidoNet Resource for VPC

  • Launch MidoNet CLI on MidoNet Cluster
    midonet-cli -A --midonet-url=http://localhost:8080/midonet-api
  • Create a tunnel-zone with type ‘gre’ (Generic Routing Encapsulation)
    midonet> create tunnel-zone name mido-tz type gre
  • Add hosts e.g CLC, NCs to tunnel-zone. If midolman services are running on the hosts with correct configuration, we should see a list hosts with the following command
    midonet> host list
    host host0 name alive true addresses,fe80:0:0:0:0:11ff:fe00:1101,fe80:0:0:0:0:11ff:fe00:1102,,fe80:0:0:0:eeb1:d7ff:fe7f:53bc,,0:0:0:0:0:0:0:1,,fe80:0:0:0:eeb1:d7ff:fe7f:53bc,fe80:0:0:0:eeb1:d7ff:fe7f:53bc flooding-proxy-weight 1 container-weight 1 container-limit no-limit enforce-container-limit false
    host host1 name alive true addresses fe80:0:0:0:ea9a:8fff:fe74:12ca,fe80:0:0:0:0:11ff:fe00:1102,,fe80:0:0:0:ea9a:8fff:fe74:12ca,,0:0:0:0:0:0:0:1,fe80:0:0:0:ea9a:8fff:fe74:12cb,,fe80:0:0:0:ea9a:8fff:fe74:12ca,,fe80:0:0:0:0:11ff:fe00:1101, flooding-proxy-weight 1 container-weight 1 container-limit no-limit enforce-container-limit false
    host host2 name alive true addresses,0:0:0:0:0:0:0:1,fe80:0:0:0:0:11ff:fe00:1102,fe80:0:0:0:ea39:35ff:fec5:7098,,fe80:0:0:0:ea39:35ff:fec5:7098,fe80:0:0:0:0:11ff:fe00:1101,,,fe80:0:0:0:ea39:35ff:fec5:7098 flooding-proxy-weight 1 container-weight 1 container-limit no-limit enforce-container-limit false
    # Add the hosts to tunnel zone
    midonet> tunnel-zone list
    tzone tzone0 name mido-tz type gre
    midonet> tunnel-zone tzone0 add member host host0 address
    zone tzone0 host host0 address
    midonet> tunnel-zone tzone0 add member host host1 address
    zone tzone0 host host1 address
    midonet> tunnel-zone tzone0 add member host host2 address
    zone tzone0 host host2 address
  • Set up local ASN for router
    # list router
    midonet> router list
    router router0 name eucart state up asn 0
    midonet> router router0 set asn 65996
  • Set BGP Peer (may change in future EUCA-12890)
    midonet> router router0 add bgp-peer asn 65000 address
  • Set BGP Network
    midonet> router router0 add bgp-network net


Install an image using and following command and start running instances with VPC!

python <(curl -sL
python <(curl -sL

Configure DNS server for Helion Eucalyptus

Helion Eucalyptus has come a long way since its inception in 2007. Now it comes with more services than ever with more features to make your Eucalyptus cloud robust and more scalable. However, it’s now at a point where configuring DNS has become a fundamental requirement, like they say, “With great power comes great responsibility.”

HPE Helion Logo

Eucalyptus services like Loadbalancing, Imaging and most importantly when you want to use multiple User Facing Services, configuring DNS is not optional anymore. Even though the title of the post is Configure DNS server for Helion Eucalyptus, but this DNS server can be used as a basic DNS server for other purposes in your data center, as well as usable with multiple Eucalyptus clouds at the same time.

Install packages for DNS sever:

yum install bind bind-utils

After installation we need to edit the file in /etc/named.conf to add zone specific information for forward and reverse lookup.

In this example, we have a forward zone called

zone "" IN {
        type master;
        file "";
        allow-update { any; };

For this example, we allow dynamic updates from any hosts, the default is deny all.

And since we have hosts in 10.17.198.x and in 10.17.199.x, for simplicity we will use the first two octet for reverse dns,

zone "" IN {
        type master;
        file "";
        allow-update { any; };

For this example, we disabled the DNS authentication,

dnssec-enable no;
dnssec-validation no;

The entire named.conf file should looks like below:

In the example above, is the host where DNS server is being configured.

Now that we have zones configured, we will need to configure the forward and reverse DNS records for the DNS server:

Here is an example what’s the forward DNS configuration for the DNS server (aoe-08-5) should looks like:

$TTL 86400	; 1 day		IN SOA (
				2011071306 ; serial
				3600       ; refresh (1 hour)
				1800       ; retry (30 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)

Now configure reverse lookup for the DNS server:

$TTL 86400	; 1 day	IN SOA (
				2011071301 ; serial
				3600       ; refresh (1 hour)
				1800       ; retry (30 minutes)
				604800     ; expire (1 week)
				86400      ; minimum (1 day)

Start/Restart named service:

service named start

At this point the DNS server should be ready to add records of other hosts.

We can update forward zone records for any host in the network with subdomain using the following command,

nsupdate -d
> zone
> server
> update add 86400 A
> send

Here is a small script that can be used on all the hosts to update forward DNS records, the script below also updates the hosts existing DNS configuration and adds the new DNS server in network script:

For adding reverse DNS records:

nsupdate -d
> zone
> server
> update add 86400 IN PTR
> send

Another snippet to update reverse DNS record or PTR record for all the hosts in the same network:

Finally restarting named service on the DNS server is not required to add/remove DNS records dynamically with nsupdate, but it doesn’t write the changes to the zone specific files until the service is reloaded.

Example reverse zone file after adding reloading named service:

Now since we want our DNS server to work for multiple Eucalyptus clouds, we will need to forward requests to specific Eucalyptus DNS services. So, basically add NS records for User Facing Services’ (UFS) hostnames with custom subdomain to forward all the requests to UFS for Eucalyptus to resolve service endpoints.

nsupdate -d
> zone
> server
> > update add 8600 NS
> send

In this example above, host has Eucalyptus DNS service running, so any request to will be forwarded to to get appropriate response.

After adding NS records for hosts running Eucalyptus DNS service (currently User Facing Services comes with Eucalyptus DNS service) and reloading named service on DNS server (, the forward zone file should like this:

Finally, configure Eucalyptus system properties to DNS on Eucalyptus:

euctl bootstrap.webservices.use_dns_delegation=true
euctl bootstrap.webservices.use_instance_dns=true

Check out our documentation or more information about Helion Eucalyptus:
Find us on irc: (freenode) #eucalyptus #eucalyptus-qa
Raise issues:

Eucalyptus FourZero (4.0)

Eucalyptus 4.0 is one of the biggest releases in Eucalyptus history with several major architectural changes. Lots of new re-engineered components and some behavioral changes have landed with this new release.

Major changes in Eucalyptus 4.0


Service Separation

This is the biggest one and probably the one many of us were waiting for a long time. From 4.0 CLC DB and user-facing services can be installed/registered in different hosts. With that said, now it is also possible to have multiple user-facing services (UFS).

UFS registration command looks like this,

euca_conf --register-service --service-type user-api --host --service-name API_110

And describe UFS command is given below,

euca-describe-services -T user-api


SERVICE user-api API_110 API_110 ENABLED 45 arn:euca:bootstrap:API_110:user-api:API_110/
SERVICE user-api API_112 API_112 ENABLED 45 arn:euca:bootstrap:API_112:user-api:API_112/
SERVICE user-api API_119 API_119 ENABLED 45 arn:euca:bootstrap:API_119:user-api:API_119/
SERVICE user-api API_179 API_179 ENABLED 45 arn:euca:bootstrap:API_179:user-api:API_179/

Object Storage Gateway (OSG)

Another attractive feature in Eucalyptus 4.0. With this new service, it is possible to use different object storage backends. For now OSG has complete support for RiakCS and WalrusBackend as object storage backends. Other object storages like Ceph should be pluggable as well with OSG, but is not fully tested.

More about Object Storage Gateway and RiakCS were discussed in previous posts.

Image Management

This is another great addition to Eucalyptus. Now image management was never been so fun than this. One important thing is, from 4.0 Eustore has been replaced with couple of other interesting commands in the toolset.

Installing an HVM image was never been easier,

euca-install-image -i /root/precise-server-cloudimg-amd64-disk1.img -n "demoimage" -r x86_64 --virtualization-type hvm -b demobucket

Another interesting fact is, now it is possible to get an EBS backed image from HVM image with just one single command,

euca-import-volume /root/precise-server-cloudimg-amd64-disk1.img --format raw \
--availability-zone PARTI00 --bucket demobucket --owner-akid $EC2_ACCESS_KEY \
--owner-sak $EC2_SECRET_KEY --prefix demoimportvol --description "demo import volume"

Run the following command to check the conversion task status,


When completed create a snapshot from the volume Id in the describe result and register the EBS-backed image.

Heads up: an imaging worker instance will appear running the conversion task is started.

There is another super handy command that will create an EBS backed image from a HVM image and run an instance with provided detail,

euca-import-instance /root/precise-server-cloudimg-amd64-disk1.img --format raw \
--architecture x86_64 --platform Linux --availability-zone PARTI00 --bucket ibucket \
--owner-akid $EC2_ACCESS_KEY \ --owner-sak $EC2_SECRET_KEY --prefix image-name-prefix \
--description "textual description" --key sshlogin --instance-type m1.small

EDGE Networking Mode

EDGE is a new networking mode which was introduced in 3.4 as a tech-preview feature. The main reason behind this networking mode is to remove the need of Cluster Controller to be in the data for all the running VMs. Also, this helps to eradicate the need of tagging VLAN packets to achieve Layer 2 isolation between the VMs. With this network mode, now there will be a new standalone component called eucanetd will be running on the Node Controller. In EDGE networking mode eucanetd running on the Node Controller maintains the networking and ensures any single point of failure.

Re-engineered Eucalyptus Console

This is one of the biggest changes that happened in 4.0. We said goodbye to the Eucalyptus Admin UI (https://<CLC_IP_address&gt;:8443), Eucalyptus User Console and welcomed the newly designed EucaConsole with the administrative features.

EucaConsole 4.0.0
EucaConsole 4.0

Tech-Preview of CloudFormation

CloudFormation!!! Yes, CloudFormation feature has been implemented and released in Eucalyptus 4.0 as a tech-preview, though the implementation is pretty well.

In the currently implementation of CloudFormation, the service does not come with other user-facing services, it needs to be registered separately on the same host with CLC/DB (EUCA-9505).

euca_conf --register-service -T CloudFormation -H -N API_11

Here is a basic CloudFormation template just to try it out right away,

  "Parameters" : {
    "KeyName" : {
      "Description" : "The EC2 Key Pair to allow SSH access to the instance",
      "Type" : "String"
  "Resources" : {
    "Ec2Instance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" }, "default" ],
        "KeyName" : { "Ref" : "KeyName"},
        "ImageId" : "emi-3c17bd33"

    "InstanceSecurityGroup" : {
      "Type" : "AWS::EC2::SecurityGroup",
      "Properties" : {
        "GroupDescription" : "Enable SSH access via port 22",
        "SecurityGroupIngress" : [ {
          "IpProtocol" : "tcp",
          "FromPort" : "22",
          "ToPort" : "22",
          "CidrIp" : ""
        } ]

The following command can be used to validate the template,

euform-validate-template --template-file cloudformationdemo.template

Then create a stack with the template,

euform-create-stack --template-file cloudformationdemo.template --parameter KeyName=demokey MyDemoStack

Check CloudFormation stack status,

euform-describe-stacks MyDemoStack

STACK MyDemoStack CREATE_COMPLETE Complete! 2014-06-04T14:02:27.38Z

Check CF stack resources,

euform-describe-stack-resources -n MyDemoStack

More FourZero

Apart from those, another big improvement was with Administrative Roles. There are now pre-defined roles for Eucalyptus admin account, e.g Cloud Account Admin, Cloud Resource Admin, Infrastructure Admin. ELB supports session stickiness, modify attributes of instances is supported and so on. Also many AWS compatibility issues have been fixed in this Fantastic release.

Installing Eucalyptus is now easier than ever. You can start with a CentOS 6.5 minimal server and get your own Amazon compatible Eucalyptus cloud.

To get started run the following command and have your own private cloud up and running,

bash <(curl -Ls

Enjoy Eucalyptus 4.0!!!

Clustered Riak CS with Eucalyptus Object Storage Gateway

In the last post, we have installed Riak CS on a single node. For production, a deployment of five or more nodes is required for better performance, reliability. Riak has a default three times data replication mechanism and in smaller deployment the replication requirement may not be met properly and also it may compromise the fault-tolerance ability of the cluster. Fewer nodes will have higher workloads.

According to the documentation:

If you have 3 nodes, and require all 3 to replicate, 100% of your nodes will respond to a single request. If you have 5, only 60% need respond.

In this post, we will use 5 nodes to create a Riak cluster for our Eucalyptus setup. Since we will be using Riak CS, we will be installing Riak CS in each node. We will also need a Stanchion server.

Overall, our setup will look like below:

a) 5x Riak nodes
b) 5x Riak CS nodes (one in each Riak node)
c) 1x Stanchion node
d) 1x Riak Control
e) 1x Riak CS Control
f) 1x Nginx server for load balancing between the Riak CS nodes

First we will install Riak, Riak CS on all the nodes,

yum install -y
yum install riak riak-cs -y

Configure Riak:

Modify the following lines from /etc/riak/app.config with the host IP address,

{pb, [ {"", 8087 } ]}

{http, [ {"", 8098 } ]},

Find the following line from /etc/riak/app.config and replace it with multi backend setup,


{storage_backend, riak_kv_bitcask_backend},


            {add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-1.4.5/ebin"]},
            {storage_backend, riak_cs_kv_multi_backend},
            {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
            {multi_backend_default, be_default},
            {multi_backend, [
              {be_default, riak_kv_eleveldb_backend, [
                {max_open_files, 50},
                {data_root, "/var/lib/riak/leveldb"}
              {be_blocks, riak_kv_bitcask_backend, [
                {data_root, "/var/lib/riak/bitcask"}

And add the following line to riak_core section in the config file,

{default_bucket_props, [{allow_mult, true}]},

Change the following line from /etc/riak/vm.args with host IP address,

-name riak@

Configure Riak CS:

Modify the following lines from /etc/riak-cs/app.config with the host IP address,

{cs_ip, ""},

{riak_ip, ""},

{stanchion_ip, ""},

We are going to have to create an admin user, modify the following line and set the value to true for now, change it back before going into production,

{anonymous_user_creation, false},

Change the following line from /etc/riak-cs/vm.args with host IP address,

-name riak@

Follow the same procedure for rest of the nodes.

Configure Stanchion:

Install Stanchion in one of the servers,

yum install stanchion -y

Modify the following lines from /etc/stanchion/app.config with the host IP address,

{stanchion_ip, ""},

{riak_ip, ""},

Modify the following lines from /etc/stanchion/vm.args with the host IP address,

-name stanchion@

Start Riak components on the server where Stanchion is installed:

riak start
riak-cs start
stanchion start

Create admin user:

curl -H 'Content-Type: application/json' \
--data '{"email":"", "name":"admin"}'

From the output save the following two variables,


In production system, you may want to change the anonymous_user_creation settings to false after creating the admin user.

From /etc/riak-cs/app.config and /etc/stanchion/app.config change the following two values with key_id and key_secret,

{admin_key, "admin-key"},
{admin_secret, "admin-secret"},

Restart both riak-cs and stanchion.

Setting up Riak Cluster:

Now we will join all the nodes. Run the following from each node (for this guide, I kept the stanchion node’s IP constant)

riak-admin cluster join riak@<node-ip>
riak-admin cluster plan
riak-admin cluster commit

Activate Riak Control:

Modify the following lines from /etc/riak/app.config with the host IP address,

{https, [{ "", 8098 }]},

Uncomment the ssl configuration and set file path as appropriate,

{ssl, [
       {certfile, "/etc/riak/cert.pem"},
       {keyfile, "/etc/riak/key.pem"}

Follow this guideline to create self-signed certificate,

Set the following to true from riak_control section,

{enabled, false},

and set username/password,

{userlist, [{"user", "pass"}

Login to the following url to access Riak Control web interface,


Riak Control
Riak Control

Install and Configure Nginx:

We will use Nginx as a load balancer between the nodes. It can be installed on any node or on an external server as well which can work as a load balancer between the Riak CS nodes.

We will be installing Riak CS Control which seems to be using HTTP 1.1 version (available from Nginx 1.1.4). So, we will be using latest stable version of Nginx on our Nginx server.

yum install -y
yum install nginx -y

Nginx config file will look like this [source],

upstream riak_cs_host {
  server :8080;
  server :8080;
  server :8080;
  server :8080;
  server :8080;

server {
  listen   80;
  server_name  _;
  access_log  /var/log/nginx/riak_cs.access.log;
  client_max_body_size 0;

location / {
  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_redirect off;

  proxy_connect_timeout      90;
  proxy_send_timeout         90;
  proxy_read_timeout         90;
  proxy_buffer_size    128k;
  proxy_buffers     4 256k;
  proxy_busy_buffers_size 256k;
  proxy_temp_file_write_size 256k;
  proxy_http_version    1.1;

  proxy_pass http://riak_cs_host;

Install and Configure Riak CS Control:

Run the following command to install Riak CS Control,

yum install -y

Modify the following lines from /etc/riak-cs-control/app.config with the Nginx server’s IP address and proxy,

{cs_proxy_host, "" },
{cs_proxy_port, 8080 },

Set the admin creds downloaded above in /etc/riak-cs-control/app.config file.

Riak CS Control
Riak CS Control

Configure Eucalyptus:

Now as usual, set the Riak CS property to use as Eucalyptus Object Storage Gateway’s (OSG) backend,


Instead of using Riak CS admin account since it has special admin privileges, we need to create a regular Riak CS account, via Riak CS Control or command line (like above) and use it for Eucalyptus.

euca-modify-property -p objectstorage.s3provider.s3endpoint=NGINX-IP

euca-modify-property -p objectstorage.s3provider.s3accesskey=ACCESS-KEY

euca-modify-property -p objectstorage.s3provider.s3secretkey=SECRET-KEY

Enjoy multi-clustered Riak CS with Eucalyptus!

Eucalyptus Object Storage Gateway with Riak CS

Eucalyptus 4.0 is the next major release of Eucalyptus. One of the exciting features of this release is Object Storage Gateways (OSG). It uses Riak CS as scalable storage backend. It also works with Walrus as storage backend. Object Storage Gateway first came out as tech preview in 3.4 release. To use Riak CS with OSG it is required to have an existing Riak CS setup.

In this post we will setup a minimal Riak CS setup to work with Eucalyptus OSG. For this demo I am using a Eucalyptus 4.0 setup from the currently available source from github. Here, we will be installing all the necessary Riak CS components on the same host that we are using for frontend, which is what we say a proof of concept setup and not recommended for production deployment.


Eucalyptus 4.0 introduces a new component Object Storage Gateway (OSG). Run the following command from Cloud Controller(CLC) to register this new component,

euca-register-object-storage-gateway --partition objectstorage --host <osg host ip address> <component name>

Most likely the OSG component status will be BROKEN at this point, until we configure Eucalyptus properties to work with Riak CS.

Riak CS installation and configuration:


Riak CS is built on top of Riak, one of the most popular open source distributed database. To install basic Riak CS we will need to install Riak, Stanchion and finally Riak CS. (Riak 1.4.6, Stanchion 1.4.3, Riak 1.4.3).

Set the user limit to a higher number,

ulimit -n 65536

Install Riak CS,

yum install -y

yum install -y riak stanchion riak-cs

Rest of the configuration steps are very straight-forward and can be found here.

By default Riak CS uses port 8080, while Eucalyptus also uses this port for http redirect. We need to change either of the ports to resolve the port conflict and get both Riak CS and Eucalyptus running on the same host.

To modify Eucalyptus port, run the following from CLC,

euca-modify-property -p www.http_port=<port>

To modify Riak CS port, change the port from /etc/riak-cs/app.config,

{cs_port, "<port>" } ,

While installing Riak CS, we created an admin user and got a similar json output with id, key_id and key_secret. These credentials can be used to access Riak Cloud Storage like Amazon S3.

    "email": "",
    "display_name": "admin",
    "name": "admin",
    "key_id": "BMNVZPO4ZXYAYEIFF9PG",
    "key_secret": "JXvFrTEx4eqirMGJnYZqvZiek7ZDema_1FM2CQ==",
    "id": "f181fac1f8d24fdeec39adbbbba5d13297aa6de056e1b26dc0c9e4a723cec7b2",
    "status": "enabled"

To use Riak CS with Eucalyptus OSG we need to modify the at least the following Eucalyptus properties,

PROPERTY	objectstorage.providerclient	s3
DESCRIPTION	objectstorage.providerclient	Object Storage Provider client to use for backend
PROPERTY	objectstorage.s3provider.s3endpoint	<riakcs_ip:port>
DESCRIPTION	objectstorage.s3provider.s3endpoint	External S3 endpoint.
PROPERTY	objectstorage.s3provider.s3accesskey	********
DESCRIPTION	objectstorage.s3provider.s3accesskey	External S3 Access Key.
PROPERTY	objectstorage.s3provider.s3secretkey	********
DESCRIPTION	objectstorage.s3provider.s3secretkey	External S3 Secret Key.

For Riak CS the objectstorage.providerclient will be s3, when using Walrus, the value for this property will be walrus.

Check this wiki link for few other optional configurable options.

Eucalyptus Object Storage Gateway (OSG) service status should be ENABLED now and ready to be used. Point your favorite S3 client to OSG and start using AWS compatible Object Storage.

You can use Eutester if you want to try out Eucalyptus 4.0 with OSG and Riak CS quickly. Setup Eutester and run the following script,

./ \
--password foobar \
--config /path/to/config \
--template-path "/templates/" \
--riak-cs-port <port> \
--admin-name <admin_user_name> \
--admin-email <admin_user_email>

The following python script can be used for quick OSG + Riak CS test,

import boto
from boto.ec2.regioninfo import RegionInfo
from boto.s3.connection import OrdinaryCallingFormat

if __name__ == '__main__':


    conns3osg = boto.connect_s3(aws_access_key_id=accesskey,


Happy New Year Everyone!

Eucalyptus Faststart 3.4.1 – cloud-in-a-vm on Fedora 19

Eucalyptus 3.4.1 is releasing soon, I mean, very soon. So, as a part of testing Eucalyptus Faststart, we used Fedora 19 box to try out Eucalyptus.

Cloud-in-a-vm, eh?

The journey wasn’t so bad, but I did have to touch couple of things that I never used, things that I did a while back and forgot, things I didn’t know and so on. But this morning our QA lead Victor Iglesias helped with the missing parts, in other words the reason of my suffering for a while.

Anyway, so, here is what I did to get a cloud running in a vm on a Fedora 19 box.

Installed Fedora 19 on a core i5 Dell Inspiron laptop. It is better to have some free space in the volume group while installing Fedora. We will be creating a 100GB logical volume (LV) for the cloud VM later on. Also, if there is unallocated space available in the HDD, we can use that space for the LV.

The first thing I did is disabled selinux from /etc/selinux/config

For this setup, I decided to install the @virtualization from the base group installer to keep things simpler.

yum install @virtualization

Since we will have to run AWS-like instances inside a VM, we need to enabled the nested-kvm feature from kvm_intel module. By default nested kvm is off in most of the systems.

Open/Create the following file, /etc/modprobe.d/kvm-nested.conf and add this line,

options kvm_intel nested=1

Reboot the system to enable this nested virtualization feature.

Ensure that we have nested kvm enabled,

cat /sys/module/kvm_intel/parameters/nested

“Y” represents nested kvm availability.

More on nested-kvm, here.

Now we have to create bridge (e.g br0) for the VMs.

Once the VM setup is complete, it’s almost time to start the Faststart VM.

For this setup we used a 100GB logical volume (LV).

lvcreate -L 100G -n "fslv" "volumegroup"

If there is only unallocated space available on the HDD but no free space in the volume group, create a partition (e.g /dev/sda3). Then create physical volume and extend volume group,

pvcreate /dev/sda3
vgextend "volumegroup" /dev/sda3

Uncomment the following lines in /etc/libvirt/qemu.conf file,

vnc_listen = ""
vnc_password = "<password>"
user = "root"
group = "root"

Restart libvirtd service,

systemctl restart libvirtd.service

Copy the faststart-3.4.1.iso to /var/lib/libvirt/images/ directory.

Create a 3 liner script or run these commands manually,

virsh undefine fs
dd if=/dev/zero of=/dev/fedora/fslv bs=1M count=1
virt-install --name fs --cpu host-passthrough --disk /dev/fedora/fslv --ram 4000 --cdrom $1 --graphics vnc,listen= --bridge br0

Make sure, to pass the iso file after –cdrom if you are running the above lines manually, otherwise pass the iso file as a argument with the script.

Connect to the instance with Fedora Remote Desktop Viewer or any other you like and follow the instruction to install Eucalyptus. For this installation I selected cloud-in-a-box to have all the component in the same box. This might take a while, so make sure you are caffeine enabled.

When the installation is completed, reboot the VM. In this case, you might have to start the VM again.

virsh --connect qemu:///system
virsh # start fs

It will now configure the Eucalyptus cloud and within couple of minutes an Eucalyptus cloud will be ready with one basic image and one load-balancer image.

Connect the Eucalyptus hybrid user console with the VMs IP,

User Credentials:
  * Account:  demo
  * Username: admin
  * Password: password

By default Eucalyptus Faststart installation creates admin and demo credentials.

Follow the Eucalyptus documentation to discover Eucalyptus more.

More on cloud in a vm:

Eucalyptus 3.3.0 in a nutshell

Eucalyptus 3.3.0, the most exciting Eucalyptus release so far is knocking on the door or perhaps it has been already released when you are reading this post.

Eucalyptus 3.3.0 has couple of most desired Amazon Web Services (AWS) features by the cloud users:

1. Elastic Load Balancing (ELB)

Needless to say, this is an AWS ELB compatible feature which is being introduced in Eucalyptus 3.3.0.

Creating a basic loadbalancer:

eulb-create-lb -z PARTI00 -l 'lb-port=80, protocol=HTTP, instance-port=80' MyElb
# output

# output
# LOAD_BALANCER	MyElb	2013-06-10T06:57:52.07Z

Register instances with Eucalyptus Elastic Load Balancer,

eulb-register-instances-with-lb MyElb --instances i-25D3415E,i-16463E17
# output
# INSTANCE i-25D3415E
# INSTANCE i-16463E17

eulb-describe-instance-health MyElb
# output
# INSTANCE	i-25D3415E	InService
# INSTANCE	i-16463E17	InService

Few other ELB operations,

# deregister instances from ELB
eulb-deregister-instances-from-lb MyElb --instances i-16463E17

# delete ELB
eulb-delete-lb MyElb

2. CloudWatch

CloudWatch is another AWS-compatible feature which is shipping with Eucalyptus 3.3.0. It enables cloud users to view, collect and analyze metrics of their could resources. It also lets cloud users to configure alarm actions based on the data from the metrics.

Enable instance monitoring,

# on existing instance
euca-monitor-instances i-25D3415E

# during instance run
euca-run-instances -k batman1key emi-90E83973 --monitor

# disable monitoring
euca-unmonitor-instances i-DB5842DC


# returns all the available metrics

# returns list of metrics with particular metric name
euwatch-list-metrics --metric-name CPUUtilization

# returns list of metrics with particular namespace
euwatch-list-metrics --namespace AWS/EC2

# returns list of metrics with particular dimensions
euwatch-list-metrics --dimensions "InstanceId=i-25D3415E"

# returns time-series data for one or more statistics of a given MetricName
euwatch-get-stats CPUUtilization \
> --start-time 2013-06-10T07:09:00.043Z \
> --end-time 2013-06-10T08:46:54.043Z \
> --period 3600 \
> --statistics "Average,Minimum,Maximum" \
> --namespace "AWS/EC2" \
> --dimensions "InstanceId=i-25D3415E"

3. Auto Scaling

Eucalyptus Auto Scaling is consists of three fundamental principles,

  1. Launch Configurations
  2. Auto Scaling Groups
  3. Auto Scaling Policies

Create a launch configuration,

euscale-create-launch-config MyLC \
> --image-id emi-90E83973 \
> --instance-type m1.small

Create auto scaling group,

euscale-create-auto-scaling-group MyASGroup \
> --launch-configuration MyLC \
> --availability-zones PARTI00 \
> --min-size 1 --max-size 3

# describe auto scaling groups

Create scale out policy,

euscale-put-scaling-policy MyScaleoutPolicy \
> --auto-scaling-group MyASGroup \
> --adjustment=30 \
> --type PercentChangeInCapacity

# output
# arn:aws:autoscaling::576514848852:scalingPolicy:c2a8f9dc-1c75-49d5-b54d-8ef87fe29e9a:autoScalingGroupName/MyASGroup:policyName/MyScaleoutPolicy

Creating scale in policy,

euscale-put-scaling-policy MyScaleInPolicy \
> --auto-scaling-group MyASGroup \
> --adjustment=-2  --type ChangeInCapacity

# output
# arn:aws:autoscaling::576514848852:scalingPolicy:a4148c27-81da-4eff-9140-cba3ba9381cb:autoScalingGroupName/MyASGroup:policyName/MyScaleInPolicy

CloudWatch Alarm

Eucalyptus CloudWatch alarm currently helps cloud users to take decisions on the resources (e.g instances, EBS volumes, Auto Scaling instances, ELBs) automatically based on the rules defined by the users based on the metrics. Eucalyptus CloudWatch alarm currently works with Auto Scaling policies.

Create alarm for scale out capacity and scale in capacity,

# create scale out alarm
euwatch-put-metric-alarm AddCapacity \
> --metric-name CPUUtilization \
> --namespace "AWS/EC2" \
> --statistic Average \
> --period 120 --threshold 80 \
> --comparison-operator GreaterThanOrEqualToThreshold \
> --dimensions "AutoScalingGroupName=MyASGroup" \
> --evaluation-periods 2 \
> --alarm-actions arn:aws:autoscaling::576514848852:scalingPolicy:c2a8f9dc-1c75-49d5-b54d-8ef87fe29e9a:autoScalingGroupName/MyASGroup:policyName/MyScaleoutPolicy

# create scale in alarm
euwatch-put-metric-alarm RemoveCapacity \
> --metric-name CPUUtilization \
> --namespace "AWS/EC2" \
> --statistic Average \
> --period 120 --threshold 40 \
> --comparison-operator LessThanOrEqualToThreshold \
> --dimensions "AutoScalingGroupName=MyASGroup" \
> --evaluation-periods 2 \
> --alarm-actions arn:aws:autoscaling::576514848852:scalingPolicy:a4148c27-81da-4eff-9140-cba3ba9381cb:autoScalingGroupName/MyASGroup:policyName/MyScaleInPolicy

# delete alarm

Set the alarm state to OK/ALARM for testing,

euwatch-set-alarm-state --state-value OK \
> --state-reason "testing" AddCapacity

euwatch-set-alarm-state --state-value OK \
> --state-reason "testing" RemoveCapacity


# output
# AddCapacity	OK	arn:aws:autoscaling::576514848852:scalingPolicy:c2a8f9dc-1c75-49d5-b54d-8ef87fe29e9a:autoScalingGroupName/MyASGroup:policyName/MyScaleoutPolicy	AWS/EC2	CPUUtilization	120	Average	2	GreaterThanOrEqualToThreshold	80.0
# RemoveCapacity	OK	arn:aws:autoscaling::576514848852:scalingPolicy:a4148c27-81da-4eff-9140-cba3ba9381cb:autoScalingGroupName/MyASGroup:policyName/MyScaleInPolicy	AWS/EC2	CPUUtilization	120	Average	2	LessThanOrEqualToThreshold	40.0

4. Resource Tagging

Resource tagging was another missing AWS feature which was not there until 3.2.2. This is a very important feature and also used by many 3rd party tools and application.

euca-create-tags vol-65803EB8 --tag "testtag"
# TAG volume vol-65803EB8 testtag

# VOLUME vol-65803EB8 2 PARTI00 available 2013-06-10T12:29:41.082Z standard
# TAG volume vol-65803EB8 testtag

5. More instance type

INSTANCETYPE	Name         CPUs  Memory (MB)  Disk (GB)
INSTANCETYPE	m1.small        1          256          5
INSTANCETYPE	t1.micro        1          256          5
INSTANCETYPE	m1.medium       1          512         10
INSTANCETYPE	c1.medium       2          512         10
INSTANCETYPE	m1.large        2          512         10
INSTANCETYPE	m1.xlarge       2         1024         10
INSTANCETYPE	c1.xlarge       2         2048         10
INSTANCETYPE	m2.xlarge       2         2048         10
INSTANCETYPE	m3.xlarge       4         2048         15
INSTANCETYPE	m2.2xlarge      2         4096         30
INSTANCETYPE	m3.2xlarge      4         4096         30
INSTANCETYPE	cc1.4xlarge     8         3072         60
INSTANCETYPE	m2.4xlarge      8         4096         60
INSTANCETYPE	hi1.4xlarge     8         6144        120
INSTANCETYPE	cc2.8xlarge    16         6144        120
INSTANCETYPE	cg1.4xlarge    16        12288        200
INSTANCETYPE	cr1.8xlarge    16        16384        240
INSTANCETYPE	hs1.8xlarge    48       119808      24000

Well, if you used Eucalyptus before, I think, the improvement is very much visible 🙂

6. Maintenance Mode:

Eucalyptus 3.3.0 also comes with the feature which many cloud administrator might be waiting for such a long time, which is Maintenance Mode.

In other words, migrating a single instance to another Node Controller or evacuating a certain Node Controller are now supported by Eucalyptus.

# evacuate a Node Controller
euca-migrate-instances --source

# migrate specific instance to another destination
euca-migrate-instances -i i-38A74228 --dest

For more information check the Eucalyptus 3.3.0 roadmap. Architectural overview for 3.3.x release can be found on githubHere is a list of new stories that are going to take place in the 3.3.0 release.

More AWS Compatibility

Eucalyptus 3.3.x is the most AWS compatible release ever. It has more API compatibility than Eucalyptus ever had. Here is couple of our ongoing work on the different AWS SDKs and open source libraries.

  1. AWS SDK for Java
  2. AWS SDK for Ruby
  3. AWS SDK for PHP
  4. AWS toolkit for Eclipse
  5. jcloud on Eucalyptus – This is comparatively newest among all, we are tracking this as a story on jira, EUCA-5671.

Eucalyptus 3.3.0 has few very important improvements on Boot-from-EBS instances,

1. Root block device is /dev/sda and not /dev/sda1
2. Allow multiple EBS block device mappings
3. No more default ephemeral disk at /dev/sdb
4. Metadata service changes

Euca2ools 3.0 is huge in Eucalyptus 3.3.x. It has been completely ported to requestbuilder. Euca2ools 3 is slim and beautiful and it works!

One interesting fact about Euca2ools from the developers,

% git diff –shortstat 2.1.3.. — bin euca2ools
432 files changed, 14973 insertions(+), 15097 deletions(-)

euca2ools 3 adds three entirely new services and tons of new
functionality to the previous version, but it still manages to weigh
in at less code than it had before.

Read more about euca2ools 3, “What’s new in Euca2ools 3” Part 1 and Part 2.

With all these new features, Eucalyptus 3.3.0 has many bug fixes as well. There are many others documented/undocumented fixes are coming in 3.3.0. Some administrator tool are also on the way to see the light very soon.

If you are interested in trying from source code, you are more that welcome to checkout Eucalyptus from the public github repository.

Some places to give you feedback:

Bug report: