Friday, February 12, 2010

How to select the best automated software testing tool quickly

Well, you know how it is. You have been testing an application for some time now. One fine day, your manager walks over to you. He tells you that the management (or your client) is interested to get the functional tests automated. Since you are very familiar with the application, you should suggest the most suitable automated software testing tool for the purpose. You may know about your application’s technology and about a number of automated testing tools. However, you may not be clear about how to do justice to the evaluation exercise without burning time over this extra work. View the video, How to Select Automated Testing Tools. Then read on.

How to select the best automated software testing tool quickly

Evaluation of automated software testing tools can be a challenging and effort-prone task. Over the years, I have successfully analyzed, compared and evaluated a number of automated functional testing tools for applications written in various technologies. Below, I will describe the process for evaluation that I think is balanced in terms of effort and results.
1. Define your requirements

The primary requirement is that the automated testing tool should work (in other words, be compatible) with the application under test. There may be other requirements such as:
a. Ease of use
b. Available documentation
c. High quality customer support
d. Cost effectiveness (i.e. affordable price of license)

See the more comprehensive list of requirements towards the end of this post. Take your pick and/ or add requirements that are specific to your project, client or organization.

The requirements are rarely equally important. This means that some requirements are more important than others. After you choose your requirements, assign a weightage to each requirement using a scale, say 1 to 10 (1 being the worst and 10 being the best).

2. List the tools

There are many lists of automated functional testing tools on the web. One example is here. It is important to cast your net wide. Otherwise, you may miss a testing tool that beautifully suits your purposes. Later, someone (your manager, your colleague or your client) may ask why you did not consider another “obvious” tool.

3. Create a scorecard

Creating a scorecard need not be complex. Just list each of the requirements on the X-axis and the tools from your list on the Y-axis. See the sample. Add more columns and rows as required.

Look up the documentation on each tool's website. You could also browse the popular software testing/ QA forums e.g. to get a feel of the kind of problems test automators generally face with a particular tool. Fill in the score (along with optional notes) in each cell. Use a consistent scale, say 1 to 5 (1 being the lowest and 5 being the highest), in each cell. You should have the raw data available at this time.

4. Analyze the scorecard

Next, you create the total scores for each tool. In order to get the total score, take the following steps for each row.
a. Multiply each score with the respective requirement weightage.
b. Add up all products in step a.

See the example scorecard with dummy scores. Note that the score for Tool1 is 4*8 + 5*5 + 2*10 = 77.

5. Try the short-listed tools

After you analyze your scorecard, you should see the tools that satisfy your requirements more than the others do. Short-list the top tools. You should select at least the top two tools (in order to give you some choice) e.g. Tool2 and Tool1 in the above table and at most the top three tools (in order to spare yourself too much effort).

It is the norm for tool vendors to offer trial or evaluation copies of the automated testing tools. Download and install the evaluation copy of each of the short-listed tools. Use each tool with your application. Create and run a few tests. Explore each of your requirements (given in your scorecard). Soon, you should have a good handle on:
a. the suitability of each short-listed tool with your application
b. your comfort level while using the tool

6. Present your results

You should create an automation demonstration with each of the short-listed tools. At least, you should create the demo for the top tool so far. Try to cover concepts related to as many requirements (from your scorecard) as you can. For example, if keyword driven testing were a requirement, it would be useful to have a keyword driven test within your demo.

With your scorecard data and your demo, you should be confident in presenting the results of your evaluation to any stakeholders.

Please fee free to use any information in this post for your purpose. Kindly let me know your thoughts on this evaluation process.

Possible requirements from the automated software test tool

• System requirements for tool installation
• System requirements for test creation
• Platforms supported for test creation (including platforms supported by using add-ins)
• Popularity (wide-spread use)
• Ease of use
• Object recognition
• Object management
• Integrated development environment
• Test debugging
• Integration with version control software
• Data driven testing
• Keyword driven testing
• System requirements for test execution
• Platforms supported for test execution
• Error recovery
• Reporting
• Documentation and training
• Customer support
• Load testing capability
• Test management capability
• Cost effectiveness (i.e. affordable price of tool and add-ins licenses)


  1. Hi, Your blog is really wonderful. I liked this article since it is very useful to me. Thanks for sharing it. Bye.

  2. Very nice!! there is so much information on this blog keep posting good information like this so that I can come back every day for some new info...
    soa testing

  3. Hi,
    How about unit testing tools, would you also use this method ?


  4. Wonderfully done. I appreciate this.

  5. This is super helpful. People forget that even the automated software tests were evaluated by live people. :) Thanks for the post, I'm a beginner and it's just what I've been looking for.

  6. Your blog is excellent, and you have provided very important information through this blog. I was looking for a similar blog for a long time that can clear my doubts. That's why I want to thank you, and I wish you to keep writing similar blogs in the future. Apart from this, if you ever need a Digital E-Learning Service
    then you can get all the information on this website.

  7. With the constant demand change in software industries, automation is necessary for every step to create the best products. The traditional methodology follows manual testing; however, with the vast data and functionality, every company is going with automation. Besides, it guarantees that all capabilities work appropriately and cover the vast majority of the likely parts. Automated Web Test Tools

  8. Avo Assure is a codeless QA testing software that helps you test end-to-end business processes. It makes software testing automation easy and quick