At Eucalyptus, if one thing we care about it is the quality of the product. Our goal is to deliver an AWS compatible software that just works. To ensure the highest level of quality we try to follow the optimum strategy possible in both Development and in QA.
We have adopted Agile Software Development model a while back and we love it. As a part of our strategy, there are several type of projects we open in Jira. After getting the green signal from Product Management, we get Epic and then after architectural discussion Story tickets are created with Sub-tasks for developers to work on. We also have Jira tickets for New Features which are basically similar to Story but comparatively smaller tasks and generally have no Sub-tasks. At this point one scrum master from each team works closely with the developers to keep track of the progress. Then comes Bugs, everyone in the company or outside the company can report bugs, of course they can open new features as well. Typically, the workflow is very simple, all the bugs becomes confirmed either by the Tier3 members or in the bug scrub meetings. Then a developer gets assigned to a bug and the bug status gets from In Progress to In QA and after successful verification it goes into Release Pending state by a Quality Engineer. At this point the bits are ready to be merged in the master branch. And YES! Everything we do is publicly visible, we love Open Source!
As Quality Engineers, our job is to be in the process where a Story or New Feature or Bug’s status becomes In QA.
Eucalyptus is a complex piece of software and like most other system application out there, it is necessary to check not only the bug or any specific feature but also the entire product and with continuous development the testing is also a continuous process. Here, I will quote from one of colleagues, Kyo Lee, “Q: When do you stop testing? A: You never stop testing.” Achieving this goal would never be possible if we would plan this to do manually (or multiply the number of employees by 4 times?). Here comes Eutester, Eutester4J and se34euca, test frameworks built for Eucalyptus, but not limited to Eucalyptus.
Eutester is a functional testing framework for Eucalyptus written in Python. It uses Boto, a python interface to Amazon Web Services to ensure the AWS compatibility. Most of the Eucalyptus services are covered with various test cases which are written on top of Eutester. On the other hand, Eutester is definitely not limited to Eucalyptus functional testing, there are scripts to help with installation of Eucalyptus, Riak CS etc. It is also used to collect logs and various configuration, artifacts from the Eucalyptus hosts to debug issues. There are test cases for both administrators and users. Also, many test cases are built to reproduce and verify bugs. All the test cases are found on Eutester github repository.
For Java developers we have Eutester4J, another test framework built with AWS Java SDK, yet another product to test AWS Java SDK compatibility a.k.a AWS compatibility. It uses TestNG to build various test suites for Eucalyptus. For dependency management, it uses Ivy to make life easier for all.
Now to test the user console we have this fantastic project that we call se34euca, which uses Selenium and automates the entire process of testing the user console.
Well, we have all these projects, but how do we achieve automation at it’s highest level? The answer is Jenkins! We use Jenkins to automate all the test cases. We took the advantage of continuous integration to automate the test frameworks. Now the answer leads us to more questions, “how/where do I install Jenkins? how do I integrate those frameworks?” Yes, to answer all those questions, we have MicroQA. It’s the home of all those frameworks. MicroQA uses Vagrant (yet another cool tool, eh?) to speed up the installation. Yes, it is even possible to test Eucalyptus cloud from your laptop.
With all these, what we never forget is Manual Testing. We do test manually, mainly when a new feature becomes available, sometimes when we verify certain bugs, manual testing is unavoidable. We use open source test management software Testlink to manage manual test cases.
Last but not least, unlike typical tech organizations, we work very closely with developers, we all make sure we understand our job which is not to count the number of bugs fixed/raised/verified, nether the number of features/test cases, rather it is to deliver a quality product.
shaon's blog 