Scenario: You are a competent tester who has just joined a company and is testing on the first project there. The application you are testing is a financial web application. During test execution, you make the following observations. Rate each observation on a scale of 1 to 10, 1 being definitely not a bug and 10 being certainly a bug.
1. The logo on the home page and other pages of the application is not the client's. For example, if the client's name were 1, the logo is that of another company, say 2. The logo in the previous version of the application is your client's.
2. There is a group of links on the pages of the application. When you click any link in this group, it opens a page with the "Not Found" error.
3. You log on as a new user into the application. You make a deposit of $ 1,000 into your account. When you visit the Transactions page, it shows you your transaction of $ 1,000. However, when you visit the Balance page, it shows your available balance as $ 2,000.
Should you go ahead and report these observations as bug reports pronto?
.
.
.
.
.
.
.
.
.
.
My advice is No. All your observations mean is that you should investigate further. For all you know, there might be perfectly reasonable explanations for these observations. For example,
1. Your client is getting this application developed for a partner organization. Or, their parent organization. Hence, the different logo. The change request is already under construction and you would get it soon.
2. The group of links will be supported by an external entity. During your test, you should just have checked if each link is correct. The external entity would populate these links before the test finish date and then all the links in the group would work.
3. There is indeed a requirement which states that when a new user makes the first deposit between $ 1,000 and $ 2,000, the client organization doubles the balance by making an equal deposit into the user's account. However, you are not aware of this requirement.
The objective of this post was to help you realize how using Heuristics may lead you along a wrong path. However, I did not state this objective at the beginning of the post because I did not want to bias you.
Testing is an investigative process. We should observe the application carefully, formulate theories if what we see is not in line with our mental model, gather more information from multiple sources and analyze and test our theories before pronouncing a discrepancy or a bug in the application.
1. The logo on the home page and other pages of the application is not the client's. For example, if the client's name were 1
2. There is a group of links on the pages of the application. When you click any link in this group, it opens a page with the "Not Found" error.
3. You log on as a new user into the application. You make a deposit of $ 1,000 into your account. When you visit the Transactions page, it shows you your transaction of $ 1,000. However, when you visit the Balance page, it shows your available balance as $ 2,000.
Should you go ahead and report these observations as bug reports pronto?
.
.
.
.
.
.
.
.
.
.
My advice is No. All your observations mean is that you should investigate further. For all you know, there might be perfectly reasonable explanations for these observations. For example,
1. Your client is getting this application developed for a partner organization. Or, their parent organization. Hence, the different logo. The change request is already under construction and you would get it soon.
2. The group of links will be supported by an external entity. During your test, you should just have checked if each link is correct. The external entity would populate these links before the test finish date and then all the links in the group would work.
3. There is indeed a requirement which states that when a new user makes the first deposit between $ 1,000 and $ 2,000, the client organization doubles the balance by making an equal deposit into the user's account. However, you are not aware of this requirement.
The objective of this post was to help you realize how using Heuristics may lead you along a wrong path. However, I did not state this objective at the beginning of the post because I did not want to bias you.
Testing is an investigative process. We should observe the application carefully, formulate theories if what we see is not in line with our mental model, gather more information from multiple sources and analyze and test our theories before pronouncing a discrepancy or a bug in the application.
P.S. The idea to write this post came after reading, ICICI Bank ATM bug … Try it!............. by Amit Jain, a member of the Software Testing Space group.
Awsome blog inder sir...... unfortunately earlier i found all the 3 scenarios as bugs :-( unless I could really go through your explanations given below...or could think at that perspective...Surely big lesson learnt is that a tester should,or in fact must go through all the processed and pending customer change requests, before we jump to actual test execution...
ReplyDeleteInder,
ReplyDeleteNice post and I completly agree with your views here. Testing is indeed an investigative process and one should not jump to conclusions in just one GO. Everything depends on the requirements OR I should say signed-off/documented requirements from the clients. Before logging on any bug a tester must investigate the scenario and also the documented requirements.
Amit
Good reminder Inder!
ReplyDeleteRelates very well to
“Don’t fix bugs unless users want them fixed.”
"A bug is something that bugs someone who matters." as seen on http://www.testthisblog.com/2010/08/your-users-dont-want-that-bug-fix.html
Respectfully I disagree with your defect identification and reporting position. Here is why with each scenario you raised.
ReplyDeleteScenario 1: The logo on the home page and other pages of the application is not the client's. For example, if the client's name were 1, the logo is that of another company, say 2. The logo in the previous version of the application is your client's.
Response 1: Assuming that the tester has the requirement stating the logo should be the clients name (1) but the requirement really should be for another company (2) then this is a requirements defect and should be logged. The defect will not count against the Development Team but it will count against the Requirements team. Each time a requirement is changed in the testing process which requires a change to the code or environment this introduces risk. Also by capturing these metrics we are measuring the accuracy and effectiveness of the SDLC process specifically with requirements in this case. Testing is not just about reporting on system readiness it is also about understanding the process and methodology that was used to build the system. Understanding the process and its failures will help to take corrective action going forward.
Scenario 2; There is a group of links on the pages of the application. When you click any link in this group, it opens a page with the "Not Found" error.
Response 2: This is would be reported as an environment defect. Which raises the question is the test environment setup correctly and what other issues are out there. If a large number of environment defects are reported then you have to ask the question how reliable was the test effort?
Scenario 3: You log on as a new user into the application. You make a deposit of $ 1,000 into your account. When you visit the Transactions page, it shows you your transaction of $ 1,000. However, when you visit the Balance page, it shows your available balance as $ 2,000.
Response 3: Why isn’t the tester aware of the requirement? Is it because the requirement wasn’t documented? If so, this is a requirement defect. Is it that the tester missed this requirement in the BRD? If so, why? Is it a training issue? A workload issue? A scheduling issue? In any case this would be reported as a Test defect. If there are lots of defects because of tester error wouldn’t you want to know and understand that? How can you improve a process if you don’t understand the root cause?
Respectfully,
TestMasterGenius