Clustered Riak CS with Eucalyptus Object Storage Gateway

In the last post, we have installed Riak CS on a single node. For production, a deployment of five or more nodes is required for better performance, reliability. Riak has a default three times data replication mechanism and in smaller deployment the replication requirement may not be met properly and also it may compromise the fault-tolerance ability of the cluster. Fewer nodes will have higher workloads.

According to the documentation:

If you have 3 nodes, and require all 3 to replicate, 100% of your nodes will respond to a single request. If you have 5, only 60% need respond.

In this post, we will use 5 nodes to create a Riak cluster for our Eucalyptus setup. Since we will be using Riak CS, we will be installing Riak CS in each node. We will also need a Stanchion server.

Overall, our setup will look like below:

a) 5x Riak nodes
b) 5x Riak CS nodes (one in each Riak node)
c) 1x Stanchion node
d) 1x Riak Control
e) 1x Riak CS Control
f) 1x Nginx server for load balancing between the Riak CS nodes

First we will install Riak, Riak CS on all the nodes,

yum install -y
yum install riak riak-cs -y

Configure Riak:

Modify the following lines from /etc/riak/app.config with the host IP address,

{pb, [ {"", 8087 } ]}

{http, [ {"", 8098 } ]},

Find the following line from /etc/riak/app.config and replace it with multi backend setup,


{storage_backend, riak_kv_bitcask_backend},


            {add_paths, ["/usr/lib64/riak-cs/lib/riak_cs-1.4.5/ebin"]},
            {storage_backend, riak_cs_kv_multi_backend},
            {multi_backend_prefix_list, [{<<"0b:">>, be_blocks}]},
            {multi_backend_default, be_default},
            {multi_backend, [
              {be_default, riak_kv_eleveldb_backend, [
                {max_open_files, 50},
                {data_root, "/var/lib/riak/leveldb"}
              {be_blocks, riak_kv_bitcask_backend, [
                {data_root, "/var/lib/riak/bitcask"}

And add the following line to riak_core section in the config file,

{default_bucket_props, [{allow_mult, true}]},

Change the following line from /etc/riak/vm.args with host IP address,

-name riak@

Configure Riak CS:

Modify the following lines from /etc/riak-cs/app.config with the host IP address,

{cs_ip, ""},

{riak_ip, ""},

{stanchion_ip, ""},

We are going to have to create an admin user, modify the following line and set the value to true for now, change it back before going into production,

{anonymous_user_creation, false},

Change the following line from /etc/riak-cs/vm.args with host IP address,

-name riak@

Follow the same procedure for rest of the nodes.

Configure Stanchion:

Install Stanchion in one of the servers,

yum install stanchion -y

Modify the following lines from /etc/stanchion/app.config with the host IP address,

{stanchion_ip, ""},

{riak_ip, ""},

Modify the following lines from /etc/stanchion/vm.args with the host IP address,

-name stanchion@

Start Riak components on the server where Stanchion is installed:

riak start
riak-cs start
stanchion start

Create admin user:

curl -H 'Content-Type: application/json' \
--data '{"email":"", "name":"admin"}'

From the output save the following two variables,


In production system, you may want to change the anonymous_user_creation settings to false after creating the admin user.

From /etc/riak-cs/app.config and /etc/stanchion/app.config change the following two values with key_id and key_secret,

{admin_key, "admin-key"},
{admin_secret, "admin-secret"},

Restart both riak-cs and stanchion.

Setting up Riak Cluster:

Now we will join all the nodes. Run the following from each node (for this guide, I kept the stanchion node’s IP constant)

riak-admin cluster join riak@<node-ip>
riak-admin cluster plan
riak-admin cluster commit

Activate Riak Control:

Modify the following lines from /etc/riak/app.config with the host IP address,

{https, [{ "", 8098 }]},

Uncomment the ssl configuration and set file path as appropriate,

{ssl, [
       {certfile, "/etc/riak/cert.pem"},
       {keyfile, "/etc/riak/key.pem"}

Follow this guideline to create self-signed certificate,

Set the following to true from riak_control section,

{enabled, false},

and set username/password,

{userlist, [{"user", "pass"}

Login to the following url to access Riak Control web interface,


Riak Control
Riak Control

Install and Configure Nginx:

We will use Nginx as a load balancer between the nodes. It can be installed on any node or on an external server as well which can work as a load balancer between the Riak CS nodes.

We will be installing Riak CS Control which seems to be using HTTP 1.1 version (available from Nginx 1.1.4). So, we will be using latest stable version of Nginx on our Nginx server.

yum install -y
yum install nginx -y

Nginx config file will look like this [source],

upstream riak_cs_host {
  server :8080;
  server :8080;
  server :8080;
  server :8080;
  server :8080;

server {
  listen   80;
  server_name  _;
  access_log  /var/log/nginx/riak_cs.access.log;
  client_max_body_size 0;

location / {
  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_redirect off;

  proxy_connect_timeout      90;
  proxy_send_timeout         90;
  proxy_read_timeout         90;
  proxy_buffer_size    128k;
  proxy_buffers     4 256k;
  proxy_busy_buffers_size 256k;
  proxy_temp_file_write_size 256k;
  proxy_http_version    1.1;

  proxy_pass http://riak_cs_host;

Install and Configure Riak CS Control:

Run the following command to install Riak CS Control,

yum install -y

Modify the following lines from /etc/riak-cs-control/app.config with the Nginx server’s IP address and proxy,

{cs_proxy_host, "" },
{cs_proxy_port, 8080 },

Set the admin creds downloaded above in /etc/riak-cs-control/app.config file.

Riak CS Control
Riak CS Control

Configure Eucalyptus:

Now as usual, set the Riak CS property to use as Eucalyptus Object Storage Gateway’s (OSG) backend,


Instead of using Riak CS admin account since it has special admin privileges, we need to create a regular Riak CS account, via Riak CS Control or command line (like above) and use it for Eucalyptus.

euca-modify-property -p objectstorage.s3provider.s3endpoint=NGINX-IP

euca-modify-property -p objectstorage.s3provider.s3accesskey=ACCESS-KEY

euca-modify-property -p objectstorage.s3provider.s3secretkey=SECRET-KEY

Enjoy multi-clustered Riak CS with Eucalyptus!

8 thoughts on “Clustered Riak CS with Eucalyptus Object Storage Gateway

  1. Nice post. You should add that best practice is to create a new RiakCS account (via RiakCS Control if desired) that will be used exclusively for Eucalyptus rather than configuring Eucalyptus to use the RiakCS admin account created during installation. The RiakCS admin account has special privileges (e.g. can see and change all other RiakCS accounts), so it shouldn’t be given out for regular use. It’s best for Eucalyptus to just use a regular RiakCS account.

  2. Thank so much for the step by step instructions. I’m following along, and I’m stuck where you start riak on the Stanchion server. I get this output:
    !!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.

    riak failed to start within 15 seconds,
    see the output of ‘riak console’ for more information.
    If you want to wait longer, set the environment variable
    WAIT_FOR_ERLANG to the number of seconds to wait.

    The riak console then shows:

    Node ‘riak@’ not responding to pings.
    config is OK
    !!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.
    Exec: /usr/lib64/riak/erts-5.9.1/bin/erlexec -boot /usr/lib64/riak/releases/1.4.10/riak -config /etc/riak/app.config -pa /usr/lib64/riak/lib/basho-patches -args_file /etc/riak/vm.args — console
    Root: /usr/lib64/riak
    Erlang R15B01 (erts-5.9.1) [64-bit] [smp:16:16] [async-threads:64] [kernel-poll:true]

    /usr/lib64/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
    Erlang has closed
    {“Kernel pid terminated”,application_controller,”{application_start_failure,riak_kv,{invalid_storage_backend,{riak_kv_app,start,[normal,[]]}}}”}

    Crash dump was written to: /var/log/riak/erl_crash.dump
    Kernel pid terminated (application_controller) ({application_start_failure,riak_kv,{invalid_storage_backend,{riak_kv_app,start,[normal,[]]}}})

    1. Found out my issue, in the /etc/riak/app.config file, I incorrectly had
      {add_paths, [“/usr/lib64/riak-cs/lib/riak_cs-1.4.5/ebin”]},
      However, the Riak CS version I had was installed in the -1.5.0 directory, so changing to
      {add_paths, [“/usr/lib64/riak-cs/lib/riak_cs-1.5.0/ebin”]},
      fixed it for me. Thanks again for the awesome write up!

  3. Ok, sorry for all the spam comments, but I’m having issues getting the Riak control up. I think I’m misunderstanding something. Under the “Activate Riak control” section you mention the /etc/riak/app.config file line starting with https, and the port number is 8098, but later when you login to the page, you use port 8069 (https://RIAK-NODE-IP:8069/admin). When I try to access it on 8098, I get an error from Firefox that the connection was interrupted. On 8069, it says it is unable to connect. Any ideas? Thanks.

    1. For HTTPS you will have to use a different port that HTTP and use that port to access the Riak control panel. So, in this case 8069 port was used in “Activate Riak control” section.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s