Interviews of Thought Leaders - Alan Page (final part)

The first part of the interview with Alan Page was read and appreciated by many. All, many thanks for your responses and questions. For me, some of the things mentioned by Alan are sure to change my approach to testing in subtle but fundamental ways. Well, here is the second and final part of Alan's interview:

7. What skills and qualities would you look for if you interview a software test engineer for the purpose of hiring?

[Alan] I interview a lot of candidates on my new team, so I can pull from recent experience. As hinted above, I look for a passion for learning – I try to find out how and when testers learn more about what they do – I want to understand what they know about testing and how they form their opinions about testing. Problem solving relates pretty well to this, and also to test investigation and debugging. I don’t ask the silly “how many piano tuners are in the world” type of questions – instead, I usually just ask test candidates to troubleshoot some random problem – e.g. an application won’t start, what do you do? This gives me some insight into methodology, problem analysis and creativity. I also look for test design (or test aptitude). For example, I may ask how to test a random number generator or a string search function. Both of these examples are simple on the surface, but have a lot of deeper problems under the covers. Good test candidates can see the whole problem quickly and get creative on testing the challenging parts (I left the details vague on purpose - don’t want to ruin the “fun” for potential candidates).

8. What advice would you give to the software test leads/ test managers working in various organizations? What should they do if they want to progress in their testing career?

[Alan] A big part of your role as a test lead or manager is to make sure that your team is performing and growing. The work you assign for your team needs to be challenging, yet achievable. As a lead or manager, you need to base your success on your team’s success – and take the heat if/when things don’t go well. I sometimes see test managers who have good ideas, but think that mandates are the only way to make change. Managers need to learn other methods of organizational change and use a variety of approaches when they want change to happen. I also encourage test managers to stay involved in the testing process. Some managers think that test management is just about management, and not about testing – these managers almost always fail.

9. Many software testing professionals find effort estimation a bit challenging. What advice can you give to arrive at a fairly accurate estimate, especially in automation projects?

[Alan] First off, as you may know, I’m not really a fan of GUI automation – but it can work if planned correctly. If you can involve development in the estimate – e.g. if programmers and testers can come up with the estimate together, you may get closer. The problem is that there usually too many “unknown unknowns” in GUI automation efforts – if you make programmers aware of this, they are more prone to add the appropriate testability features (e.g. an object model) to the UI so it can be tested in a less fragile manner.

Beyond that, normal estimation practices apply – break tasks down into the smallest unit possible – preferably a day or less. This makes it much easier to know when you’re behind (or ahead of) schedule before it’s too late, but more importantly, makes you take the time to fully understand the task in front of you.

10. We know that you handle sizable responsibilities. In addition, you are a frequent conference speaker and author. Could you give us some tips on how do you manage your time and energy so successfully?

[Alan] I don’t work 80 hour weeks – at least I don’t think I do. I have a job, I have stuff I do that’s related to my job (blogging, speaking, writing, etc), and I have my personal and family time. But I don’t really segregate them – they all sort of merge together into the 16 or 17 hours I’m awake every day. Sometimes I take care of personal stuff (I try to run 3-5 miles every day) or blog during the day, and I usually work from home for an hour or two at night after the kids go to sleep on whatever is important.

As far as managing my time, it’s very scrum-like. I have a “product backlog” of everything I do in my outlook tasklist. It includes my work projects and personal things (like “do my taxes”, or “set up conference with teacher”). Some items are recurring. My scrum team of 1 does 1-day sprints – each morning, I go through everything on the todo list for today and prioritize. Some items get deferred if I know they aren’t critical for that day, and sometimes I’ll add things. I don’t get through everything I plan every day, but I put the unfinished items back in the backlog and re prioritize the next day. It may sound a little silly (I don’t actually think of it as scrum-like when I’m doing it), but it’s the only way I can stay on top of everything I’m doing.

I frequently remind people (and myself) that there is always time to do whatever you want – you just need to make the right choices to make that time for yourself. You may think, for example, that you don’t have a single spare moment the entire week – but if you were so sick you couldn’t get out of bed for two days, I bet you’d be able to “catch up” within a week. As long as you’re healthy, take the time to invest in what you think is important, and your task list will work itself out. (note – I blogged about this here if you need clarification or elaboration)

Of course, knowing when to say “no” is also critical. I hate to turn down writing or speaking requests, but I still have to do it more often than I’d like.

11. Do you think that you made the right choice in selecting software testing as a career?

[Alan] Absolutely – I never thought I would be a software tester this many years, but I can’t imagine doing anything else. At one point in my career, I briefly entertained the idea of becoming a sw developer, and was even assigned some feature work and bug fixing on a few different occasions at Microsoft (I have written functionality that shipped with Windows and Windows CE). On both occasions, however, I realized that my heart was in testing (for example, I enjoyed investigating and fixing the bugs far more than writing features from specifications. I’m sure I’ll be a tester until I retire – and I hope that’s a long time from now, because I’m still having fun and enjoying the job a lot.

12. Finally, would you like to share any other advice/ wisdom with us readers?

[Alan] –Sure - The biggest advice I can share is to focus on learning and develop your own thoughts and opinions. I see too many testers chasing after silver bullets and shiny objects – blindly trusting testing advice from tool salesmen and self-proclaimed testing experts. If you haven’t learned enough and practiced enough to form your own opinions and build your own toolbox of testing approaches, you have little chance of success.

Many thanks to Alan for taking his valuable time to pen his thoughts on pertinent questions from all of us in the software testing space.

Please comment if you have any related questions. I will do the same.

Automatic test data generation

Among other things, you need test data to execute tests. You are required to design the test data to drive your test cases. If the test data is not already listed with your test cases or you run tests in an exploratory manner, you need to design the test data on the fly. Generating your own test data takes away your valuable time. Generating test data by hand is especially time consuming if you have to run automated tests (e.g. functional tests or performance tests) which need a large quantity of distinct test data.

Of course, if the production data is available, you may get its copy and use it for test. However, this is not a very wise approach. Using live data in test may expose confidential customer information to unauthorized people. In fact, acts like HIPAA address the security and privacy of data.

The good thing is that you need not rely on generating test data by hand. Several tools are available that can help you create large quantities of test data based on the constraints or conditions specified by you. Automated test data generation has the following benefits:

Fast	If the number of test data items is large, automated test data generation may be substantially faster than creating the same test data by hand.
Accurate	Since test data generation is a tedious task, you are likely to make mistakes. Test data generated by a tool can be much more accurate.
Can use built-in algorithms	The test data generator may implement algorithms to generate correctly formatted special data like postal codes and email addresses. Therefore, you need not look up the algorithms for generating the special test data yourself.
Create both valid and invalid test data	Just as you can create valid test data, you can also create invalid test data by specifying incorrect constraints. Further, by specifying the quantity of the valid and invalid data, you can control the percentage distribution between valid and invalid data.
Data exportable to multiple formats	The automated test data generators usually provide options to either generate the test data directly in the desired format (e.g. Excel, CSV or SQL) or export to the desired format.
Create a professional look	Since automated test data generators use their own databases to source data, the test data added/ edited in your application looks professional. This way, you can avoid test data that looks odd (e.g. Name as aa or abab12). This is important in a way because the test data used by you is stored in your application and reported in your bug reports. If your client or management looks at your bug reports, you would feel better if the test data looks good (e.g. Name as Jackson or Burke).

Even though an automated test data generation may offer so many benefits, you should remember that it is just another software. Like other software, it may or may not have all the features you need or always generate the correct data. However, if you are willing to try automated test data generation, you could be boosting your productivity or the productivity of your team.

Note: I said "among other things" since sometimes it is necessary to populate an application with test data before the application can even function (fully).

Test Strategy: How to create a powerful test strategy?

Let us begin with discussing what is a Test Strategy? As you know, Strategy means using the resources at one's disposal to achieve the desired objectives effectively. Test strategy means the plan that details how the testing objectives would be met effectively. See the video, How to create a  test strategy or read on.

Like any other strategy, the test strategy may be defined at different levels. You may have a test strategy at the organization level, at a program level or at a project level. The thing is that it may not be called a test strategy at the organization/ department/ program level; it may just exist as a management policy or as a part of a governance plan. At the project level, the test strategy may just be a part of the test plan. Further, depending on the nature of the project, the test strategy defined at the project level may or may not satisfy the test strategy outlined at a higher level.

The advantages of an effective test strategy are:

1. It mitigates the risks to objective testing.

2. It helps focus on different aspects of the system under test (by using distinct test phases such as unit testing, integrating testing and so on).

3. It provides clarity on the required people, procedures, tools and infrastructure.

4. It can be used to communicate the test approach to the various stakeholders (clients, project/ test managers, developers and testers) in the project.

How to create a powerful test strategy?

If you want to define a powerful test strategy for your project/ program/ organization, you should consider the following tips:

1. Before you define your test strategy, you should research the client(s) and the end user(s) with respect to the their needs and expectations from the application(s).

2. Your test strategy should aim at satisfying the test objectives alone.

3. Practically speaking, you may need to create two versions of your test strategy depending on the situation - one for communicating to all and a detailed/ tailored one for agreement with the key stakeholders.

4. You have the option of having different kinds of focus in different test phases. However, do not go over-board with defining too many test phases with overlapping effort. In other words, economize with the test phases. Define only the ones which you think are really required.

5. Your test strategy should be totally customized according to your specific situation. What has worked for other companies or what has worked for your company in the past may not apply to your current situation.

6. You should consider the required test environment while designing your test strategy. You should at least have one test environment that is the same as (or is a replica of) the production environment. Further, your test strategy should assume the most common test environment configurations for the most part.

7. Your test strategy should define the testing tools (test/ defect management systems and/ or automated test tools). It is best if you have evaluated the testing tools yourself. If not, base your decision on reliable (vendor-agnostic) and popular user reports.

8. You should look at any defined test processes (e.g. entry/ suspension/ resumption/ exit criteria for test, process to execute test cases and process to report a defect) to examine their feasibility in your situation. Identify the required test processes that will be re-used, modified or created from scratch.

9. Identify the data that will be recorded, measured, analyzed and reported to show the progress of testing.

10. Challenge all assumptions while you design your test strategy. Provide safeguards if any of the prior assumptions prove incorrect later.

11. Finally, remember that a powerful strategy is never static. Monitor the results obtained from the test strategy and do not hesitate in making the desired modifications to your test strategy.

See also:
Example test strategy
How to evaluate software testing tools faster?
How to have your team members excel and grow?
Risk management in software testing
How to encourage testing within your organization?

Interviews of Thought Leaders - Alan Page

As promised in my post, Interviews of Thought Leaders in Software Testing, here is the first interview. This interview is with Alan Page. Alan is the Principal SDET at Microsoft Corporation. In the past, he has also played the role of Director, Test Excellence at Microsoft. Along with being a frequent conference speaker and author, Alan is a prominent thought leader in software testing. Alan also writes on his blog at angryweasel.com.

Though I had been reading about Alan and his thoughts before, my interaction with him began in the year 2008. I respect his vast knowledge, professional responses and helpfulness. In this interview, you can read Alan's thoughtful and detailed responses to what I hope are pertinent questions from all of us in the software testing space. Who know, some of Alan's replies may subtly lead you to even greater testing successes.

1. We know that you are a very senior person in the area of software testing. What roles have you played in your software testing career so far and what other roles would you like to play in the future?

[Alan] I don’t know that I’m senior – perhaps just highly exposed. I started testing before I knew what testing was – I was hired to perform product support for a music software company and was notified the first day that I was also the (only) software tester. When I joined Microsoft a few years later I tested configurations and user interface for the Windows 95 networking components. My role on the windows 98 team was primarily API testing – with a slant towards compatibility testing. Later on, I ended up working a lot with the kernel and drivers and ended up owning those areas of Windows CE as a test lead for just over a year. I’ve always felt I was a stronger performer as an Individual Contributor than as a lead or manager, so when I stopped my stint as a lead I moved into a Test Architect role (which was really just a fancy title for a senior non-manager tester at Microsoft). As a TA, I functioned as a technical leader for the CE test team – acting as a go to person, a reviewer, and a strategist for the team. I spent about five years in Microsoft’s Engineering Excellence group – the first two as a technical instructor and pseudo-consultant, and the last two and a half as the Director of Test Excellence where I was sort of the overseer for Testers at Microsoft – in charge of technical training, long term strategy, etc. In my new role on the Office Communicator team I don’t test a specific part of the product – instead I try to identify everything I can that will help make the test team more effective and improve overall software quality.

In the future – at least for the next few years, I see more of the same – perhaps with larger teams or tackling even harder problems.

2. What are the different kinds of tests that you have done or led/ managed/ directed?

[Alan] You name it, and I’ve probably done it (although not always successfully).

3. Could you tell us about a particularly tough challenge that you faced in your testing career? How did you resolve it?

[Alan] My toughest challenge is always my next challenge – I’m always looking for new challenges. However, I’ve probably had a few particularly interesting challenges. The first that comes to mind was when I was first asked to join the Windows ME team.. The debug version had a ton of great error checking that enabled application developers to find a slew of programming errors early. The problem was that the debug version of Windows had been ignored for completely a year and wouldn’t even boot. In addition to kernel and driver testing, I was asked to “fix” debug windows (and own debugger development as well). I had no idea what I was doing, yet somehow happened to make it all work. It was one of those times when I felt I was completely over my head, but somehow managed to figure it out. I’ve always been able to learn quickly and to hold a bunch of disparate information in my head and somehow put it together in the right order.

4. What is the one achievement in your testing career that you are most proud of?

[Alan] It’s probably writing (and finishing) How We Test Software at Microsoft. Sometimes I still can’t believe I took on that project, but I’m glad I did.

There are plenty of things I wish I would have done differently in that book, but I’m still proud of getting it done and that people like what we came up with.

5. How do you think the software testing space is transforming now? What specific software testing roles/ jobs would increase in the future? What skills and knowledge do you think will be required by software testers in the coming years?

[Alan] Software is getting more interconnected and complex rapidly – the challenge of testing this interconnected and complex software is growing rapidly.

It’s really not enough to find surface level functionality bugs anymore – the important bugs are lurking within the connection points where functions, modules, and features connect. Testers are learning that they need to have a strategy for determining where to target testing, and what types of testing to perform to get the maximum value from their time.

Testers spend a lot of time looking at data – today, it’s most often bug data and pass rates, but many testers also look at coverage reports, code churn or complexity metrics. The problem is that most testers aren’t data analysts or statisticians, and aren’t really very good at figuring out what action to take from data. The result is that the data is either ignored, or misinterpreted. I think the business intelligence role will become more important in the future of test – without meaningful analysis of the massive amount of test collateral generated by software teams, test will never be able to catch up with the growing complexity of future applications.

6. What qualities and abilities do you like in the younger software testing professionals? Which qualities you do not like in the younger crowd?

[Alan] You’re making me feel old :}!. Beyond anything else, I think a passion for learning is the most important attribute of a tester. I also think it’s important to point out that “passion” has to be demonstrated rather than stated. In other words, I admire testers who seek new knowledge and actively look for ways to understand how to apply that knowledge – rather than those testers who state random bits of new information and look for ways to dismiss that knowledge without taking the time to understand it.

Well, this was the first part of our interview. Answers to the following questions are still to come:

* What skills and qualities would you look for if you interview a software test engineer for the purpose of hiring?

* What advice would you give to the software test leads/ test managers working in various organizations? What should they do if they want to progress in their testing career?

* Many software testing professionals find effort estimation a bit challenging. What advice can you give to arrive at a fairly accurate estimate, especially in automation projects?

* We know that you handle sizable responsibilities. In addition, you are a frequent conference speaker and author. Could you give us some tips on how do you manage your time and energy so successfully?

* Do you think that you made the right choice in selecting software testing as a career?

* Would you like to share any other advice/ wisdom with us readers?

Be sure to check out Alan Page's replies to these questions shortly. Is there any other question that you would like to ask Alan?

[Update on 29 Apr 2010] You can now go to the second and final part of this interview.

If you are interested in test automation, you have probably come across the term, Test Automation Framework, for example, as I explain in my Selenium Python Page Object Model Tutorial. First, see a Test Automation Framework in action in my Selenium Unit Testing Tutorial. Then read on.

A Test Automation Framework commonly consists of the implementation of certain concepts and code that gives a jump-start or provides support to one or more test automation projects. Before you design or enhance your Test Automation Framework, you should identify the objectives that you want to achieve with it.

Test Automation Framework features

The desired features of your Automation Framework may include:

General features
1. Re-use of existing code by writing code once and invoking it elsewhere
2. Consistency of test automation implementation by defining standards
3. Tool independence by implementing your framework as separate software
4. Ease of framework installation
5. Ease of framework maintenance
6. Ease of upgrading the deployed framework
7. Types of testing supported by the framework
8. Supported operating systems
9. Supported browsers

Features related to test design
10. Simple to learn and use
11. Common operations already built-in the framework
12. Help documentation with examples
13. Minimal changes to the test automation in case the application changes :)
14. Integration with version control systems and test management systems

Features related to test execution
15. Ability of the framework to run on multiple systems
16. Support for test configuration e.g. application name, server name/ URL and test cases to be executed
17. Support for scheduled execution
18. Supported invocations of the framework e.g. by GUI or by command-line
19. Support for invoking and closing/ shutting down the application(s) under test
20. Ability to connect to myriad test data sources and collect test data
21. Fast performance
22. Robustness
23. Error recovery e.g. by dismissing unknown windows, retrying a failed operation with/ without configurable delays, proceeding to the next step/ test case and even resetting or re-starting the application(s) under test
24. Integration with defect tracking systems
25. Clean up test data in the application(s) under test after completing each test

Features related to test reporting
26. Logging states and results
27. Reporting test results (in one or multiple formats e.g. Excel or HTML) with metrics such as number of test cases in the test plan, test cases executed, test cases passed, test cases failed and the time taken to execute the test cases
28. Distributing test results e.g. by publishing to web or email

When you estimate the effort it is going to take to build the desired features in your Automation Framework, you may find that the time available is not sufficient. You should then prioritize the features and build the most important ones first. Start small. As your Automation Framework is used in test automation projects, you may learn about the problems faced while using your framework. You should then prioritize the fixes and enhancements and build upon your existing Automation Framework accordingly.

Interviews of Thought Leaders in Software Testing

I have been planning to interview prominent figures in the space of software testing for quite some time now. These are the people who are leaders in software testing. They have excellent command on the topics of software testing and have been helping and influencing software testers across the world.

The first interview in this series would be with Alan Page of Microsoft Corporation. If you do not know Alan already, he is the Principal SDET at Microsoft Corporation. In the past, he has also played the role of Director, Test Excellence at Microsoft. Along with being a frequent conference speaker and author, Alan is a prominent thought leader in software testing. Alan also writes on his blog here.

Watch out STS for a candid interview with Alan Page.

How to set up a new testing department?

Many companies choose to set up a testing department instead of (or in addition to) allocating testers permanently to projects. The rationale to do so include:

A. Availability of a more comprehensive skill set
You may have access to a larger skill set (e.g. manual as well as automated testing, functional as well as performance testing and knowledge of a number of different business domains) if you have multiple members in your department.

B. Better management of the testers' time
Testers are commonly under-utilized when the code is being written and are very busy when the application is under test. Pooling testers together provides an opportunity to level their task assignments better.

C. More learning opportunities for testers

The testers in a testing department may get chances to work on multiple projects. This provides them opportunities to enhance their existing skills as well as develop new ones.

The membership of the test department may vary from just one member to several hundred members. A large testing department may be called a testing organization, a testing practice or a testing center of excellence in the company.

If you are a manager tasked with setting up a new testing department, here are the things that you should consider for your plan:

1. Objectives of your test department

You should be very particular about eliciting and understanding the objectives of your testing department. Everything else should map with these objectives. You should know the scope of activities that the testing department would perform. Examples of objectives could be:

    a. Support pre-sales activities for new clients from a testing perspective

    b. Provide testing for the flagship product(s) of the company

    c. Provide testing on company projects

    d. Deliver testing projects

    e. Undertake research on testing topics

2. Team members required

You should estimate the team size (number of full time/ part time team members) required depending on the size of expected work as per the objectives of your department. You should also list the skills (technical, business and soft skills), experience and/ or testing certifications that should be available with the team members.

3. Tools required

In order to work effectively, your department may need tools. Such tools may include test management tools, automated testing tools (e.g. functional testing tools and performance testing tools) and project management tools. You may not want to purchase any tool licenses ahead of time. However, you may want to evaluate several tools of each type so that you may quickly make a decision to buy license(s) later.

4. Test processes required

In order to enable your team to work consistently and quickly, you would need a number of core testing processes. Examples of the testing processes include:

    a. Test effort and duration estimation

    b. Test methodologies for various types of testing

    c. Test automation process

    d. Defect management process

5. Internal processes required

In addition to the core testing processes, your department would require internal processes for supporting the team members. Examples of these processes include:

    a. Planning, scheduling and tracking the testers time

    b. Induction of new team members

    c. Career development and appraisal process

    d. Training

    e. Team building

6. Infrastructure required

Depending on the objectives of your department, you may need to procure and establish the required infrastructure (hardware including special devices, software including tools and frameworks and network including devices and bandwidth). You would create, modify or optimize the required test labs from this infrastructure available to you.

Once you have planned your testing department, you should go about obtaining or creating the necessary assets. However, you should keep the following tips in mind while doing so:

I. The objectives of your department may change after it is established. This usually happens when your department shows promising results with some work. Therefore, you should analyze your department in light of the new objectives and re-plan accordingly.

II. It takes vastly different durations of time to obtain different types of assets. For example, you may purchase a tool license in a day but you may need to wait a month to recruit a new team member or get a new server and you may take 6 months to develop your own estimation process. Keep this in mind.

III. A testing department commonly has numerous assets. Therefore, you should plan long-term with the goal of utilizing each asset to a reasonable extent.

Now, you should know that setting up a new test department requires establishing its objectives, obtaining the team members, tools and infrastructure and creating or reusing core testing processes and internal processes. I hope that your test department is able to fulfill its objectives and more.

Windows applications test automation: Free functional testing tools

I have seen several requests (especially on LinkedIn) from software test professionals looking for free automated testing tools that work with Windows applications. Here is a list of tools that you may want to look at and evaluate for your test automation project:

1. pywinauto

2. AutoIt

3. AutoHotkey

4. TestSmith

5. QAliber

What is the best way to test screen access based on roles?

Many applications implement role based security and role based access. This means that when the user accesses the application, the application determines the role of the user first. The application shows only those screens to the user which are meant for the particular role. The application may even hide or show certain parts or sections of a particular screen depending on the role of the user. For example, an application may show only the public screens to an unauthenticated user. If the user logs in as a member, the private screens of the particular member become available to the user. If the user logs in as an administrator, the private screens of the particular administrator as well as the application administration screens become available to the user.

If you have to test the screen access of your application, you should go about it systematically. Here is the approach that you can use:

1. List the various kinds of access in the application.
Examples of access include View (just see the information listed on a screen), Read (see more information of a data item), Drill down (see the sub-items of a data tem), Add (add a data item), Update (edit a data item), Delete (remove a data item) and Comment (add further information to a data item).

2. List the various roles in the application.
You should refer the requirements documentation of your application to list the roles in your application. Examples of roles are User, Super user, Customer, Staff, Manager and Administrator.

3. List each screen (and each section within each screen) in your application.
You should refer the requirements documentation as well as the design documentation of your application to list each screen in your application. If your application implements role based access on sections (parts) of any or all screens, you should group all the relevant screen sections under the screen name.

4. Map the roles to the screens listing the access in each case.
Finally, you should map roles to the screens. In order to do so, simply list all roles on one axis and all screens on the other. Since the number of roles is commonly far less than the number of screens, you may want to list the roles horizontally and the screens vertically for ease in viewing or editing your mapping. See the example above.

5. Ensure that your mapping is complete.
Ensure that you have listed each access, each role and each screen (with each section that is accessible separately) in your application. Ensure that the mapping is present between each role and each screen/ screen section.

How to be a software testing expert in your company?

You know how it is if you work in a company with multiple software testers. The management's perception differs for different members of the software testing team. A few people are held in high regard. They are the ones who are first introduced to new or existing clients. Their names come up first when the membership of a new project is discussed. Then there are others, who despite having good knowledge and a respectable experience are seemingly always sitting on the sidelines.

Use the following tips and lift your image in your company. You can find all these tips and much more in the video, How to become Software Testing Expert in your team or company.

1. Update the list of your skills.
You learn new skills all the time while working in your project. Instead of waiting for the company performance evaluation/ appraisal exercise, make a list of the new skills that you have acquired. Submit the list of your skills to be updated in your company's records.

2. Participate in meetings actively.
You probably attend a number of project meetings/ division meetings or company meetings. Participate actively in these meetings. You may start by preparing questions for the meeting and then raising them in the meeting. If you are comfortable, you may provide informed comments to the topic of discussion in your meeting.

3. Ask your client/ manager to recommend you.
There may be occasions when your client or your manager is very happy with the work done by you. You should ask to be recommended but do it tactfully. If you receive a written recommendation for your good work, you can share it with others and also keep it for future reference.

4. If possible, try to get into a high visibility project.
We know that of all the projects being executed, a few projects are more important (e.g. in terms of size, cost, duration, returns or prestige). Naturally, the important projects (and the team members working in it) are monitored closely and more often.

5. Create generic assets in your project.
You create a number of assets (e.g. documents, templates, processes, procedures and utilities) in your project. Instead of creating an asset just for your project, you may create a generic one with a little more effort. The advantage of creating a generic asset is that it may be used in other projects (of course, with the permission of the owner of this intellectual property).

6. Learn and try to solve a few problems in other projects.
If you know about any urgent problems in another projects and are able to invest the time to help solve them, just go ahead.

7. Keep an eye on the upcoming projects.
If you come to know about the projects in the pipeline, you may be able to get the information to prepare yourself for the same. Company meetings and the office grapevine are some ways to get information on the upcoming projects. If an upcoming project and your skill set match, ensure that the relevant resource manager knows about this match.

8. If possible, get on site experience.
Working directly at the client's site is a great way to know about the client's business, their processes, their working culture and to develop personal rapport with the client's personnel. If you have a successful on site experience, do not be surprised if the client asks for you by name for the next project.

9. Develop rapport with the other managers in your company.
You probably already share a rapport with the manager you report into. Introduce yourself to the other managers in your company. Let them know about your experience, knowledge and skills. A good way to do this is to talk to them after-hours (when things are not so hectic).

10. Write in your company's newsletter.
If your company runs a newsletter, they probably keep looking for contributors for every issue. You may research and write a piece on software testing.

The above tips may require a little more effort from your side or may require you to change your approach slightly. However, they can be very effective in changing how you are perceived in your company and in elevating your image.

What are the challenges that you can face in your testing career?

There are challenges that you may face as you progress in your testing career. By a challenge, I do not mean a (pressing) problem that you face temporarily in your project. Rather, a challenge comprises of a complex (and usually, a group of) long term problem(s) that we may face while working for a particular client or while working with our current organization.

In my opinion, we encounter a challenge when our thought process and the thought process of an important stakeholder (e.g. the client, your management or your team) do not match. Here are some examples:

Challenges due to your client or management
1. Your client or management thinks about testing as a cost which should be minimized. As is usual with cost approvals, you need to build a solid case for every piece of testing that you think would help.
2. Your client or management thinks about testing as a very predictable process. You are expected to prepare very detailed test plans (akin detailed project schedules) and explain every discrepancy between the planned and actual activities.
3. You receive little input on your planned tests. However, if bugs escape to the delivered application, you are grilled about your test plans.
4. There is limited or no budget for testing tools or support tools. You have to spend a lot of your valuable time to manually perform certain repetitive tasks.
5. The test environment is too restrictive (e.g. it is available only for limited times, it is also used by other teams or it is ridden with stability problems). You end up spending a lot of time waiting for the test environment to be available or need to check it constantly for problems.

Challenges due to your own abilities
1. You have been asked to use a tool that you know little about. Therefore, you face two problems. First, you need to learn the tool. Then, you need to find ways to effectively use the tool in your project.
2. You are expected to use a communication style that is different to your natural communication style. For example, if you prefer a verbal communication style, you have been asked to create detailed written reports in your project. Or, if you prefer to take time to carefully draft written reports, you are now expected to provide constant verbal updates.

Challenges due to your team
1. You have been added to an established team and you are finding it hard to integrate and collaborate with the team quickly.
2. Your team members are less skilled in certain testing activities. Therefore, you find that the bulk of the required work has fallen in your lap.
3. Your team members are casual in their approach to testing. You now need to double check the completeness and accuracy of their work.
4. Your team is located afar and you lack instant updates from them.

Challenges due to another team or individual(s)
1. Word is spread unofficially about your poor performance or the poor performance of your team. This discourages you.
2. Due to any reason, another team (e.g. the development team, the deployment team or the IT team) does not effectively collaborate with you or your team.

Realizing a challenge and coming to terms with it takes time. It requires your long-term commitment to resolve a challenge. Then, you need to put in constant effort to either ease the challenge or resolve it partially or completely. The good thing about handling challenges is that they force you to solve a complex problem and you emerge even more confident than before. If nothing else, talking about a challenge and how you resolved it makes a great story. What is a big challenge that you have faced in your testing career so far?

Team Management: Why your offshore test team cannot perform (well)?

Companies have been off-shoring work due to cost advantages and unavailability of local personnel as well as for mitigating risk and due to other practical compulsions. However, over the years I have heard about numerous instances when the offshore test team was unable to perform well. The issues in the performance of the offshore test team have the potential of breaking down the entire communication between the on-site team and the off-shore team. The presence and severity of these issues depend on

a. The level of commitment of your on-site management to the success of the off-shore team

b. Your level of knowledge of the people, processes and tools off-shore

c. The off-shore team's level of knowledge about the on-site people, processes, tools and working pressures

d. The intensity of communication and transparency between the on-site and the off-shore teams

e. The level of support given by the off-shore management to the off-shore team

f.  Factors attributable to individuals

Depending on the above factors, your off-shore team may or may not face issues with their performance. In fact, I have experienced off-shore test teams that perform well as an integral part of the overall test organization. Regardless, you should be aware of the potential issues with the performance of the off-shore test teams. Let us see what these issues are.

Issues due to poor on-site management

1. Without realizing it, you do not give due credit to the work done by the off-shore team. Your on-site team members duplicate this approach.

2. Due to personal reasons, the on-site team members do not share information readily with the off-shore team.

3. The on-site and off-shore team members do not travel to the other location. Therefore, they only have a working relationship but no rapport with each other.

4. You micro-manage the off-shore team, wanting to know about their daily plan in detail and asking them to update you after finishing every task. This leaves them with lesser time to focus on their core tasks.

OR

Due to your other commitments, you do not review the work done by the off-shore team at all or in a timely fashion.

Issues due to distance

5. Due to differences in time zones, your off-shore team does not participate in your important meetings. They miss out important information updates and come to know about them (much) later, may be after there is a problem. Due to lack of timely communication, they do not realize the urgency of situations developing in the project.

6. If you wish to pass on important information or discuss something urgently, you have limited options other than to write an email or wait to call the next day.

7. Using email as the primary means of communication forces you and your off-shore team to compromise on the completeness of information exchanged.

Issues due to language differences

8. The off-shore team cannot understand your verbal or written updates or instructions completely or in the spirit intended by you.

9. You have trouble understanding their communication e.g. verbal status updates or defect reports.

Issues due to company differences

10. The rigor of the processes followed by you and them is totally different. For example, you are agile but your off-shore team has been following a regimented approach in their projects so far.

Issues due to skill level differences

11. Some team members off-shore have little knowledge of testing.

12. The off-shore test team knows little about your application/ product/ service. Due to this, they focus on the wrong areas of the system under test or miss obvious problems in test.

13. They are not familiar with the tools used by the on-site organization.

Issues due to differences in working style

14. Unlike you, they do not work independently. They wait to be told and do only what you have specifically asked them to do, nothing more.

15. Whereas your working hours are flexible, their working hours are fixed due to local constraints.

Issues due to poor off-shore management

16. The off-shore team does not have a strong and communicative leader. You have a vague idea of the problems faced by the off-shore team or their plan or progress.

17. The work done by the off-shore team is not reviewed locally but submitted directly to the on-site team. This leads to problems that you need to resolve at both ends.

18. Your off-shore team is afraid of losing your work. Therefore, they seek approval from the on-site team members constantly thereby, losing the ability to work on their own with confidence and taxing the time of the on-site team members.

19. Your off-shore team members are simultaneously working on projects other than yours, without your knowledge.

20. Your off-shore team membership keeps changing. This forces everyone concerned to put in their efforts from scratch to bring the new individual up to speed.

Issues due to individuals

21. An off-shore team member is facing a personal issue affecting their productivity. You are not aware of this.

22. The off-shore team has one person who is the incorrect role model for the others. This intensifies the problems of the off-shore test team.

However, the situation is not all bad. You may experience success with your off-shore test team if you have the foresight to identify the potential issues, mitigate them and are motivated to make the relationship work over a long term. Build an off-shore team with hand-picked skilled, reliable, stable and motivated individuals. See what issues exist with your team. Make them aware of the issues and resolve each issue with the help of your off-shore test team. If your off-shore team is like most others, they would be more than happy to make the necessary changes to work up to your expectations.

What issues have you faced in working with an off-shore/ on-site team? Comment on the issue faced by you.

Stress Testing

Prem Phulara had commented on my earlier post, What knowledge do you need to have in order to do good or best performance testing asking me to explain stress testing.

This post is dedicated to Prem. In this post, we will discuss stress testing.

Stress testing is a special type of performance testing. It is used to stress (or in other words, put a high load on) the system to determine its highest operational capacity. Stress tests establish the highest load that the system is able to withstand while still being operational (and not throwing too many errors). The additional benefit of stress testing is that it exposes the resources that run out the fastest when the load on the system approaches its maximum sustainable value. It is common to design a series of load tests with increasing loads to find out the maximum sustainable load.

Here are the tips to perform stress testing correctly but quickly:

1. Before you begin work on stress testing your system, you should be aware of the goals of your stress test. For example, your goal could be to determine the maximum throughput of your application for a given business transaction/ workflow on the given infrastructure or your goal could be to determine the maximum number of concurrent users with a given workload and the given infrastructure.

2. It is crucial that you identify the correct performance testing tool for your stress test. Otherwise, you would be forced to make compromises in your tests. See How to evaluate automated software test tools quickly?

Also see Open Source Software Test Tools.

3. You would need to model test script(s) according to the business transactions specified or implied in your goals.

4. If your scripts are correct, a large number of business transactions would execute during your tests. Therefore, ensure that you have a large set of test data that does not exhaust during the test. You may consider sourcing any existing test data from your application's database. Another way of quickly generating some test data is to use Microsoft Excel to extend a series. There are also free test data generators available on the web.

5. As with other tests, you should have an isolated (as far as possible) environment for conducting your tests. Unless you stress test a system on a stand-alone machine, your test environment may consist of the client machine(s), the load generator machine(s), the server(s) (web server(s), application server(s), database server(s) and so on) and the connectivity between the above. You would need your performance testing tool installed on at least one machine. You would set up your test, run it and analyze the test results from this machine. Depending on your choice of the performance testing tool, other load generator machine(s) may or may not require the full performance testing tool installed on them.

6. When you set up your first load test, you should model it realistically. For example, you should add the required scripts to the test, set up the specifications of the Virtual Users (e.g. initial VU, ramp-up, constant load, ramp-down and distribution of VU assigned to each script etc.) and give the run-time settings (e.g. test duration, time lag between iterations, network bandwidth distribution of the VU, their browser distribution etc.). You would really be limited by the features provided by the chosen performance testing tool.

You should specify a low load in your first load test. This is when you could check if your script(s) work correctly (e.g. the script reads the test data, it accesses the system and interacts with it, it is able to send data to the system which is accepted by the system and is able to send/ receive data to/ from other script(s) as required). If you have multiple scripts in your test, you could check if the scripts run in the correct sequence or in parallel, as specified. When your test is running, you should check the performance of each machine in your

test environment e.g. check that the server(s) and load generator(s) have become busier but not to their limit. On Microsoft Windows machines, you may use the PerfMon (Start > Run > PerfMon) tool to add your desired counters and monitor them during the test. See other tips in the post, Performance testing using HP LoadRunner software.

7. After establishing that your load test is valid, you should increase the load systematically. Do not go "big bang" and deploy the load that you think is the maximum sustainable load. For example, if you think that your system may support up to 1000 concurrent Virtual Users, start with a load test with 100 VU and keep on adding 100 VU every 5 minutes. You may find that your system supports 800 concurrent VU but crashes with a load of 900 concurrent VU. After resetting your test environment, you should then start with a load test with 800 VU and keep on adding 10 VU every 30 seconds. Then you may find that your system supports 850 VU but crashes with a load of 860 concurrent VU. Repeat the last test at least another couple of times to confirm your test result.

8. If you recall, I mentioned that the additional benefit of a stress test is to identify bottlenecks or resources that run out the fastest when the load is increased. You should analyze the test results of several of your last tests to find out about these bottlenecks. Maybe it is the network which is clogged, or your web server runs out of available memory. Confirm your finding by repeating the last few tests. Your system's capacity may be improved by increasing the resource that runs out first. Ensure that you include the details of the important bottlenecks observed during your stress test in your test report.