Test effort estimation is a skill required of a Test Lead or a Test Manager. However, test estimation is not a skill that one can learn quickly. It requires understanding of several key concepts and practice. In this post, I will explain what test effort estimation is, point you to your existing knowledge of estimation and provide you the key concepts that you can use in your estimation. First, view my video on Test Estimation techniques with formula example and Questions and Answers. Then read below.
First of all, we should understand what we mean by software test effort estimation. Test effort estimation is answering two basic questions about testing:
I. What will be done?
II. How much effort would it take?
There are other questions e.g. I. Who will do what? II. When will they do it? III. How will they do it? but these questions are not related to effort estimation but to planning and scheduling.
Even if you have not estimated test effort before (having relied on the effort estimates given by the client or your project manager), keep in mind that you do effort estimation on a regular basis. Let me explain. Do you recognize the following situations?
1. You are appearing in an examination. The duration of the examination is 3 hours and you have to answer n questions. You average the time available for answering one question while leaving out certain time for revision at the end. You look at the questions. Some questions are easy for you but some are not. You reserve less time than average for answering the simple questions and more time than average for the difficult ones.
2. You have to attend a job interview. The interview is at 10 a.m. You estimate the time it would take you to reach the interview venue, say 1 hour. You add some time e.g. 30 minutes for delays like traffic snarls. You estimate some time, say 30 minutes for collecting your documents and some time, say 30 minutes for dressing up. This means that you would need to wake up no later than 7:30 a.m. that morning to reach your interview venue in time.
3. It is the beginning of another day at work. Your manager has given you 20 test cases to execute today. In addition, you need to complete the annual self-appraisal form. You estimate that it would take you 1 hour to complete your appraisal form. Out of 8 hours of your work day, you have 7 hours remaining. You reckon that you need to execute a test case every 21 minutes (7 hours X 60 minutes / 20 test cases).
If the above situations look common to you, it means that you already do effort estimation even if you do not consciously recognize it as such.
a. Size of the system
It would take longer to test a larger system. In some projects, it is possible to know about the size of the system in terms of Function Points, Use Case Points or Lines of Code. You should take the size of the system into account when estimating the test effort.
b. Types of testing required
Sometimes, it is important to perform multiple types of testing on the system. For example, other than functional testing, it may be necessary to perform load testing, installation testing, help files testing and so on. You should create the effort estimates for each type of testing separately.
c. Scripted or exploratory testing
It may be feasible to only execute test cases or do exploratory testing or do both. If you intend to do scripted testing and do not have test cases available, you should estimate the time it would take to create the test cases and maintain them. Scripted testing requires test data to be created. If the test data is not available, you should estimate the effort it would take to create and maintain test data.
d. "Non-testing" activities
Apart from creating and executing tests, there are other activities that a tester performs. Examples include creating test logs/ reports, logging defects and entering time in the project management tool.
e. Test cycles
By a test cycle, I mean a complete round of testing (build verification testing followed by attempted execution of all test cases followed by all defects logged in the defect tracking system). In practice, one test cycle is not sufficient. You should estimate the number of test cycles it would take to promote the system to the client or production.
Now, let us understand the various approaches that you can use for test effort estimation. You may choose any of these approaches for your estimation. However, in my opinion, a combination of multiple approaches works best (by best, I mean that the effort estimates are close to the real actual efforts). In any case, you should be aware about the following approaches:
1. Use historical data from other projects
This approach is useful when you have effort data available from earlier projects which are very similar to the current project. For example, this approach is useful in the case of long-running projects where the test effort data from previous releases is readily available.
2. Your organization's approach
Your organization may have their custom approach to estimate test effort in projects.
3. Delphi method
This is useful when you have a number of experts knowledgeable in the testing to be done. The experts estimate separately and then their estimates are consolidated.
4. Use your own expert judgment
This approach is useful to arrive at a rough test effort estimate quickly.
5. Software size based approach
If the size of the system is available and the formula to convert software size to test effort is available, this approach may be used.
6. Activities based approach
This approach is useful if you can list the activities required. This approach may be used Top-Down (listing the high level activities and breaking them down to lower level activities) or Bottom-Up (listing the individual activities and combining them to higher level activities). Using this approach in the Top-Down manner is better since you can control the level of detail in your effort estimate. Remember to consider activities for each type of testing, any test cases or test data need to be created, the "non-testing" activities and the multiple test cycles.
As I mentioned before, you may choose any approach to do your test effort estimation. However, using at least two approaches is better. This way, you can compare the test effort estimates and address any obvious problems. Whatever hybrid approach you choose, you should document the assumptions made by you in the estimate.
Once you have arrived at the test effort estimate for your project and have convinced the stakeholders that it is a reasonable estimate, it does not stop there. You should track the actual progress in your project constantly to see if it is in line with your test effort estimate. You may find that some of your assumptions were not correct. You should revise your assumptions and your approach in line with your observations.
Continue to use your refined test effort estimation approach across test cycles and releases. In time, you should have a good estimation approach available with you.
Refer the Q and A related to software test effort estimation. These guide you during your test effort estimation and in your discussions with project stakeholders. In addition, these make for great interview questions.
First of all, we should understand what we mean by software test effort estimation. Test effort estimation is answering two basic questions about testing:
I. What will be done?
II. How much effort would it take?
There are other questions e.g. I. Who will do what? II. When will they do it? III. How will they do it? but these questions are not related to effort estimation but to planning and scheduling.
Even if you have not estimated test effort before (having relied on the effort estimates given by the client or your project manager), keep in mind that you do effort estimation on a regular basis. Let me explain. Do you recognize the following situations?
1. You are appearing in an examination. The duration of the examination is 3 hours and you have to answer n questions. You average the time available for answering one question while leaving out certain time for revision at the end. You look at the questions. Some questions are easy for you but some are not. You reserve less time than average for answering the simple questions and more time than average for the difficult ones.
2. You have to attend a job interview. The interview is at 10 a.m. You estimate the time it would take you to reach the interview venue, say 1 hour. You add some time e.g. 30 minutes for delays like traffic snarls. You estimate some time, say 30 minutes for collecting your documents and some time, say 30 minutes for dressing up. This means that you would need to wake up no later than 7:30 a.m. that morning to reach your interview venue in time.
3. It is the beginning of another day at work. Your manager has given you 20 test cases to execute today. In addition, you need to complete the annual self-appraisal form. You estimate that it would take you 1 hour to complete your appraisal form. Out of 8 hours of your work day, you have 7 hours remaining. You reckon that you need to execute a test case every 21 minutes (7 hours X 60 minutes / 20 test cases).
If the above situations look common to you, it means that you already do effort estimation even if you do not consciously recognize it as such.
How to estimate testing efforts?
Next, let us see the factors that you need to consider before you do test effort estimation:a. Size of the system
It would take longer to test a larger system. In some projects, it is possible to know about the size of the system in terms of Function Points, Use Case Points or Lines of Code. You should take the size of the system into account when estimating the test effort.
b. Types of testing required
Sometimes, it is important to perform multiple types of testing on the system. For example, other than functional testing, it may be necessary to perform load testing, installation testing, help files testing and so on. You should create the effort estimates for each type of testing separately.
c. Scripted or exploratory testing
It may be feasible to only execute test cases or do exploratory testing or do both. If you intend to do scripted testing and do not have test cases available, you should estimate the time it would take to create the test cases and maintain them. Scripted testing requires test data to be created. If the test data is not available, you should estimate the effort it would take to create and maintain test data.
d. "Non-testing" activities
Apart from creating and executing tests, there are other activities that a tester performs. Examples include creating test logs/ reports, logging defects and entering time in the project management tool.
e. Test cycles
By a test cycle, I mean a complete round of testing (build verification testing followed by attempted execution of all test cases followed by all defects logged in the defect tracking system). In practice, one test cycle is not sufficient. You should estimate the number of test cycles it would take to promote the system to the client or production.
Now, let us understand the various approaches that you can use for test effort estimation. You may choose any of these approaches for your estimation. However, in my opinion, a combination of multiple approaches works best (by best, I mean that the effort estimates are close to the real actual efforts). In any case, you should be aware about the following approaches:
1. Use historical data from other projects
This approach is useful when you have effort data available from earlier projects which are very similar to the current project. For example, this approach is useful in the case of long-running projects where the test effort data from previous releases is readily available.
2. Your organization's approach
Your organization may have their custom approach to estimate test effort in projects.
3. Delphi method
This is useful when you have a number of experts knowledgeable in the testing to be done. The experts estimate separately and then their estimates are consolidated.
4. Use your own expert judgment
This approach is useful to arrive at a rough test effort estimate quickly.
5. Software size based approach
If the size of the system is available and the formula to convert software size to test effort is available, this approach may be used.
6. Activities based approach
This approach is useful if you can list the activities required. This approach may be used Top-Down (listing the high level activities and breaking them down to lower level activities) or Bottom-Up (listing the individual activities and combining them to higher level activities). Using this approach in the Top-Down manner is better since you can control the level of detail in your effort estimate. Remember to consider activities for each type of testing, any test cases or test data need to be created, the "non-testing" activities and the multiple test cycles.
As I mentioned before, you may choose any approach to do your test effort estimation. However, using at least two approaches is better. This way, you can compare the test effort estimates and address any obvious problems. Whatever hybrid approach you choose, you should document the assumptions made by you in the estimate.
Once you have arrived at the test effort estimate for your project and have convinced the stakeholders that it is a reasonable estimate, it does not stop there. You should track the actual progress in your project constantly to see if it is in line with your test effort estimate. You may find that some of your assumptions were not correct. You should revise your assumptions and your approach in line with your observations.
Continue to use your refined test effort estimation approach across test cycles and releases. In time, you should have a good estimation approach available with you.
Refer the Q and A related to software test effort estimation. These guide you during your test effort estimation and in your discussions with project stakeholders. In addition, these make for great interview questions.
Good article
ReplyDeleteA very informative post. I have learned many new things in your article.
ReplyDeleteThanks for sharing this post. Keep it up!
Thanks!!!! Very goog article!!!
ReplyDeleteTest point analysis and Development/Testing ratio which is usually 40% of total development estimate are some others.
ReplyDeleteRegards,
Dheeraj Saxena
Nice Post,Inder.Great job.
ReplyDeleteHi,
ReplyDeleteI had a query regarding resource estimation.
Consider this example, there are 500 test cases and 30 days to execute them. Here we need to calculate how many resources are required to complete the task.
Here the duration is fixed and resources is a variable.
Kindly let me know what formula needs to applied here.
my mail id shobhadev@gmail.com
@Shobhadev,
ReplyDeleteFirst, we need to estimate the effort for executing the 500 test cases. 30 days and the number of resources would be required later to estimate resources/ schedule the test run. So, here are a few ways to estimate the effort for running 500 test cases.
a. How long does it take to execute one test case on average (have data for executing at least 10 test cases).
b. Use approach a. but divide the test cases into categories, say Highly complex, Complex, Medium, Low, Trivial.
c. Provide sample (at least 10 test cases) to three testers. Ask them to estimate the effort, then extrapolate.
d. Use effort data for executing the same (!) or similar test cases in other projects.
Apart from executing the test cases, you should also estimate the effort for other required tasks:
1. Review and updating the test cases
2. Setting up test environment
3. Generating/ re-using test data
4. Defect logging and tracking
5. Admin tasks - meetings, status reporting, timesheets.
Then, your estimate will be more realistic. Also, you should have checkpoints to determine if the test run is proceeding as estimated or taking more/ less effort and re-estimate.
Best,
Inder P Singh
good article ... thanks for sharing your valueable knowledge.
ReplyDeleteInder P Singh,
ReplyDeleteAwesome explanations.
- Rudraneel
ReplyDeleteHey, nice site you have here! Keep up the excellent work!
Function Point Estimation Training
Can any1 share Sample documents with complete testing process on how to prepare test plan from SRS & test effort estimation? Actually, I need one example to write test plan from SRS or UseCase dcouemnt & prepare effort estimation from the same. kindly send to luckyhumain@rediffmail.com .
ReplyDeleteHi Indar,
ReplyDeleteRecently i appeared for interview of Test Lead positition and told the above estimation techniques. But the interview was not convinced. He was insisting about estimation techniques with metrices, tools etc. It will be greate if you can provide the info in these aspects.
@Tushar,
DeleteThat's the thing with people. They think in accordance with what worked for them. This does not mean that there are not other techniques that are not useful.
Anyway, the interviewer's words seem to look for 2. Organizational approach and 5. Software size based approach. He probably wanted you to talk about estimation tools (used by the company) and estimation models (on which these tools work). The way these work is that you set up the project data (WBS) in the estimation tool. The tool applies your chosen estimation model (using factors derived from historical data metrics or set up by expert judgment)upon the data and generates the estimates with +- % maximum expected variance between estimated effort and actual effort. Perhaps at its simplest, the tool can be an MS Excel spreadsheet with macros (using formulae). Once you set up EITHER the software components size details (such as screens, their fields, their complexity, their calculations, database calls, interfaces etc.) OR the activities data (WBS), it calculates and generates the estimates using the formulae set up.
Make sure that you go through my post on software testing estimation questions for more information. Its available at http://inderpsingh.blogspot.in/2013/01/EstimationQuestions.html
Thank you.