Eucalyptus HA and lock files

Eucalyptus comes with its own High Availability (HA) feature from version 3.0 and with the passage of time, it is becoming more and more stable with every release.

With recent release (3.2.0 and onwards), it has got few behavioral changes, which of course makes it more robust. To describe it in one sentence, “Lock files are to prevent data loss during HA failover”. So, lock files are to ensure that Cloud Controller (CLC) database synchronization happens properly and in any case it does not end-up in split-brain situation.

In the recent versions, Eucalyptus introduces two kinds of lock files, disabled.lock and partitioned.lock.

This file is generated to enforce the correct order of starting CLC services. So, if you stop the disabled CLC disabled.lock file is generated, then you stop the enabled CLC and try to start the disabled CLC, it won’t start, unless the cloud administrator enforce by deleting the disabled.lock file manually.

On the other hand this file is generated basically to protect split brain situation. It is to prevent two CLCs to go into enabled state. When ever there is a network outage between CLC <-> CLC, the partitioned.lock file is generated. If we see more closely, when the database becomes inaccessible, the partitioned.lock file is generated.

When the CLC fails to start because of disabled.lock or partitioned.lock Eucalyptus populates logs in cloud-fault.log. This basically represents that Eucalyptus cannot proceed by itself any further and it requires manual involvement of cloud administrator.

These are the basic use cases of both lock files. partitioned.lock is little more aggressive than disabled.lock. It actually has few more use cases, which I am yet to figure out though. In brief, when the database is killed on the enabled CLC a partitioned.lock can be generated or when the public interface is down on the enabled CLC it generates partitioned.lock file, it happens even if it’s a non-HA setup. Another case would be, if the enabled CLC cannot talk to it’s database and it finds another database on a different host Eucalyptus generates partitioned.lock.

This may not be all, the code is being improved everyday, but this is the last known behavior of the lock files.

Comments and suggestions are most welcome.


One thought on “Eucalyptus HA and lock files

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s