Tuesday, March 16, 2010

What knowledge do you need to have in order to do good or best performance testing?

Performance testing deals with design and execution of tests that examine the performance of the application under test. The application's performance is measured using a number of metrics such as application's response times, application throughput and concurrent users. Many software testers struggle when they begin performance testing/ load testing. This is because performance testing requires familiarity with a number of special concepts as well as proficiency in certain special skills. However, the good news is that you can learn the required concepts, develop the required skills and deliver results in your performance tests successfully. The purpose of this post is not to define the key terms used in performance testing but to introduce them to you. You can search these terms on the web and build your knowledge. Definitely see this video on Load Testing and Performance Testing Questions and Answers.
1. Performance testing tools (commercial, open source and custom) There are numerous performance testing tools available publicly. Some tools are commercial and the others are open source. The full-featured tools provide the functionality to create test scripts, add test data, set up tests, execute the tests and display the results. If the performance testing tool has not been chosen yet, you should evaluate the tools according to your project requirements as per the evaluation process described here.
If you have the time and technical skills, you may even create your own simple tool to help in performance testing.
2. Profilers
Profilers are tools that measure data related to the application's calls or the resources (e.g. memory) used by the application when the application is running.
3. Virtual Users
If your application supports multiple users concurrently (at the same time), you should test your application's performance using multiple users. The users modeled by the tool are called the Virtual Users.
4. Key business transactions Your application may allow the user a large number of work flows or business transactions. All the work flows/ business transactions may not be important in performance testing. Therefore, it is common to test using only the important or key business transactions. Refer or solicit your application's performance requirements for guidance in this regard.
5. Workload Workload is the load on the multi-user applications in terms of virtual users performing different business transactions. For example, in the case of a social networking application, 50 virtual users may be searching contacts, 40 of them may be messaging and 10 may be editing their profiles, all within a period of 30 minutes.
6. Isolation of test environment In order to get results with confidence, it is critical that your test environment is used only for the purpose of the performance test. This means that no other systems or users should be loading the test environment at the time of the performance test. Otherwise, you may have trouble replicating (or even understanding) your test results.
7. Modeling (script and test) You should script the key business transactions as realistically as possible (model) using the performance test tool. You should also design the tests with realistic test settings (e.g. virtual users ramp-up, user network bandwidth, user browser and so on) and the model workload.
8. Test data It is common for the test scripts to be executed numerous times during one performance test. Therefore, you may need a large amount of test data that does not exhaust during the test. Once a performance test finishes, the application may be full of dummy test data entered by the test. You may need a means of cleaning up this data (for example, by re-installing the build).
9. Server configurations You should be aware of and control the server configuration (CPU, memory, hard disk, network bandwidth and so on). This is because the application performance depends on the server resources.
10. Network configurationsYou should know about the protocols used by your application. You should also know about load balancing (in case, multiple servers are used by your application).
11. Client configurations You should know the common client configurations (in terms of CPU, memory, network bandwidth, operating system, browser and so on) used by your users. This would help you model realistic tests.
12. Load generators Depending on the load that you need to generate during the test, you may need one or more load generator machines. One tool may need fewer resources (CPU usage, memory usage etc.) per virtual user and another tool may need more resources per virtual user. You should have sufficient load generation capacity without maxing out your load generator(s) in any way.
13. Performance counters During the test, you should monitor the chosen performance counters (e.g. % Processor time, Average disk queue length) on your load generators as well as the application's servers. You should choose the performance counters so that you may come to know about the depletion (or near depletion) of any important resource.
14. Response time You should be aware that the application response time includes the time it takes for the request to travel from the client to the server, the time it takes the server to create the response and the time it takes for the response to travel back to the client.
15. Monitoring During a performance test, you should be monitoring the test progress, any errors thrown as well as your chosen performance counters on your servers and load generators.
16. Results Analysis After the completion of a performance test, you should spend time on analyzing the test results. You should check if the test created the required virtual users, generated the required load and ran to completion. You should check the errors thrown during the test. If you see any unusual results, you should form a conjecture to explain those results and look at the data carefully to either accept or reject your assumption.
It takes practice to be adept at analyzing performance tests well.
17. Reporting You should be comfortable with reporting performance test results. It is common for the performance test reports to contain present/ past metrics as well as charts.
If you make some effort, it is not difficult to educate yourself with the knowledge required for performance testing.