Web Performance

The Interview with Jim Kandler

I met Jim Kandler at the 2004 StarEast Show in Orlando a few weeks ago. We sat down for lunch and Jim told me all about his new defect detection model. Jim is still defining his model and he needs help from the industry. Read this interview to learn more. Jim runs a consultancy called Medical Software.

John: Jim, please tell us something about yourself, your background?

Jim: I've been an engineer all my life, got into testing and software quality because I was always breaking software. Colleagues would come to me, tell me they thought they had their application working, and I would break the application within 5 minutes. That's why I got into this crazy field. I've specialized in medical products; I got into it because the field really benefits a lot of people, an altruistic sort of thing. I now run a freelance consulting business.

John: When we spoke at StarEast you told me that you were building a model to determine where testing time should be concentrated in the software development process to identify the most bugs. I understand you started to develop this model because you wondered if the many testing myths about testing were actually true. What have you discovered in your research so far?

Jim: I learned of a few things that happen, the model shows you where defects get injected. It becomes obvious that the bugs are getting injected at the beginning of a project, the bugs are then being found in testing. The faults were all injected at the requirements and coding stages. The model gives you this different view. The defects are not injected at the testing stage.

John: So you are saying that there was a myth that defects were injected in the testing phase.

Jim: Yes, people think defects are injected at the testing stage. If a requirement were described incorrectly, the requirement would be wrong. Design and coding were then enacted with poor requirements.

People can do testing on requirements, that's what some of the test driven design is trying to do. Test-driven design is a way of prototyping your requirements, whenever they build out a prototype they are actually proving their models. They could bring a testing person to provide some more rigorous testing of those models.

Again, regarding my model, I found some modeling of defect detection rates. AT&T did some work years ago, through software requirements engineering. They found when they built these models, your defection rate reduces over time, it becomes harder and harder to find defects, the efficiency drops as you do more testing. The greater the level of testing the more expensive it becomes to do testing. I might spend 2 hours to remove 10 bugs, another 4 hours to find another 10 bugs, 6 hours for the next 10 bugs. The cost goes up.

I found at the 35% bug detection level, the costs of bug removal become quite high. So intuitively, go back to your defect detection model, test throughout the process, and keep to 35%. Inexperienced development practitioners wonder why they cannot get out of the system- testing phase.

There are always more defects than you remove. Every application has defects; you cannot get rid of all of the defects. I am trying to refine the model, and at some point, I will want to try and run the model against several different companies (Contact Jim here if you want to get involved.)

John: What has been your experience with companies adding additional functionality to commercial automated testing tools?

Jim: No first hand experience, I have not used any one tool long enough. I noticed a lot of them have poor reliability. On some projects we send back testing tools, the tool was un-reliable. This still tends to be a true, many of testing tools are more poorly built than the applications you are trying to use them against. We ended up finding the tool was unusable for us, we thought we were testing the application, but there was something wrong with the tool. If anyone has expectations for quality, it's a QA person. A QA person would be very upset if they have to use an unreliable tool.

John: What functionality would you look for in a good automated performance-testing tool?

Jim: I think most have the functionality that is needed for today's testing environment; to be able to do table driven testing, take some raw data, drive a table driven scenarios. Many of testing tools do secure protocols, most of the big ones allow you to do some recording and embed that in your scripts for web performance testing. The tools need to be doing some error handling, informing the user of errors, if the pages are not being returned the users needs to know that.

The other good tools allow you to coordinate multiple engines, so you can launch your scripts against multiple engines.

I did find a nice feature on one load test tool recently; the tool had a distribution function, allowing you to proportionally distribute the functionality based on the proportions you want. For example, you may want to have 20% of your users, conduct a search, 20% do other things, and 60% of users log in and sit at the home page. With three different scenarios, you can set up a distribution % per function.

John: Where do you see testing tools technology heading?

Jim: Testing tools have been a silo application; there are many silos in software development, and they hold all kinds of data. A number of the testing tool companies are starting to interconnect their tools, Mercury has Testdirector, allowing you to manage test cases, link to requirements, actually integrates the tools across the process, keeps you from generating unique silos that don't allow you to communicate.

I see a lot of people who are upset with buying tools that are just silos, at Stareast we saw a lot of this going on; we saw this across applications that communicate with one another.

John: Lastly, what are the trends in software development and testing?

Jim: People are trying to test more in the requirements phase, taking a look at rapid application development. A continuous review of deliverables is the process they use for development. Lint and syntax checkers are common. People are now starting to use context checkers, similar to spell checkers and grammar checkers. Context checkers are taking this testing up a notch, something that is cost effective.

John: Thanks Jim for some great answers to my questions.

The Interview with John Rosky

I recently met John Rosky, a QA testing professional; he agreed to answer my questions about the testing industry. John has been in QA since 1996, he has worked at Lotus development, GMIS, Micro Systems Lotus Development Corp./IBM and lastly a Portsmouth, NH-based subsidiary of Micros Systems, Inc. John has been working in the software development industry since 1987, so about 16 years.

John C: Why would you use an automated testing tool as opposed to developing in-house testing scripts?

John R: Most of my experience has been in the manual testing area; I have only used automated tools in the last two years. The reason why most people use automated tools is because they don’t want to reinvent the wheel. If a company has already put the time and effort into developing a tool, why not use that tool.

Certain tools, the windows GUI based testing, that's the easiest thing to test. A lot of the dirty work has been done for you, but if the issue is more difficult, and as the test becomes more complex, you may have to put more effort into making a good test.

John C: How have you used automated testing tool reports in the past? For what purposes did you need to develop the reports?

John R: Only used Mercury's LoadRunner product, we used it primarily for scalability. We also used the analysis tool as part of LoadRunner, which generates its own set of reports, then took data from those reports and extrapolated the data, and we used the analysis tool as part of Loadrunner.

John C: Where do you see testing tools technology heading?

John R: In the past few years, I am seeing a lot more java based testing tools. Personally, I think they could go almost in any direction, the testing tools per say are a very good thing. The tools are being developed to test more complex programming issues such as API. Now that more developers are using them, a bigger development audience has been created; tool vendors appear to be filling every void.

People want to get software products out to market faster, and if they see they can decrease the testing and development time, then they can improve quality and get the product out of the door quicker. For example, if you have five manual testers testing a large application, it would take 4 months to test an application, to test the same application with an automated testing tool, you can probably use the automated testing tool to do the testing in a week, basically you can complete a lot more testing in a shorter amount of time.

However, in my experience that's not exactly what happens every time, that’s the best scenario. Typically there is a learning curve, a cost, tools are not cheap, not a silver bullet, you have to be very realistic.

In my previous job, my manager had used WinRunner at another company, she tried for 2 years to convince management to buy the product, but the price scared them off, they wanted to know if the tool would ever give a good ROI. We were a small development shop part of a larger company and keeping within a low budget was top priority. We were already paying a developer; they thought: why not create a homegrown testing tool.

John C: What would you like to see developed by the tool vendors?

John R: I guess that I would speak specifically about Loadrunner, I thought it was very effective and useful, but it was also cumbersome to use and to learn.

I would want a simpler tool from the testing industry. Maybe I am asking for something that cannot happen, at the same time I liked how comprehensive LoadRunner was, but it was cumbersome to use initially. I guess, on a more generic sense, I would like to see a tool that does not require so much set up or potential development person intervention, a testing person who is not developer can pick up and use.

John C: We were chatting about the requirements of custom development houses and Independent Software Vendors to demonstrate software scalability to their clients in the clients’ environment. Can you recall any examples of such projects in your career and what were the pitfalls and advice you'd have for customers in handling that issue?

John R: In my experience, it was a custom application house; our customer was like an ISV. We had a centralized database server application; the clients were all over the world. The client was a huge hotel chain. The hotel chain had a distributed environment; every hotel had a server in every hotel. We were centralizing the application.

My testing involved the application and the database; we wanted to find out if we could scale the application from one hotel to 500 hotels. We tested the application to determine if it could handle so much processing, then bump up the number of hotels 20% and then bump up more. In our case, the existing hardware would scale up to 250 hotels.

The customer did not understand how challenging the project was going to be. They thought they could run a few tests, and keep bumping up the load. However, there were more problems and time involved in scaling the application than anticipated.

John C: Thanks John for answering all of my questions, I really appreciate you taking some time out to discuss these web performance issues.

The Interview with Robin Goldsmith

In an effort to understand the testing industry, explore ideas about the software development process and develop some interesting content, I am going to start conducting interviews with people in the software and testing industry for the Web Performance blog. Robin Goldsmith of Go Pro Management was kind enough to become the first Web Performance interviewee.

John: Robin, please tell us something about yourself, your background?

Robin: What you see is what you get: stunningly good looking, and I have been in the computer business since computers were working on wind power. I had a very technical background but also the good fortune to get involved with management consulting and thereby gain a management perspective too. I have been involved with Quality Assurance since before it was called QA. I’ve also been involved in outsourcing since the early days.

In the mid 80's I developed and taught what I believe was the first and only course on buying software, I’ve been involved in software acquisition from every angle. I’ve developed software for sale, assisted buyers and sellers, evaluated software, bought it, supported it, and have been the victim of what was bought. I’ve been very fortunate to be involved with a very well-known client in one of the first and largest efforts to outsource software to India.

John: Why would you use an automated testing tool as opposed to developing in-house testing scripts?

Robin: Any tool vendor, by their very nature and concentration, is able to develop proficiencies above and beyond what a normal company could develop in-house when a tool is only part of what they are doing. In addition, there really is no reason to reinvent the wheel, to coin an expression.

Having said that, I was a systems programmer, and I developed a lot of tools. I developed them because there were situations where there was no tool available, or the available tools didn’t do what I needed.

The other thing is I certainly appreciate there can be considerable issues involved with using and adapting the various commercial tools. I also feel that when you consider there are not sufficient technical people simply to use existing testing tools and develop a good testing environment, the chances that a customer has someone on staff with time and skill to develop a tool is unlikely and probably not the best use of their time.

John: Why do think there is a lack of skills in the industry?

Robin: I think the testing world is not attracting people. A lot of organizations feel that testing is not as important as programming. For example, during the technology boom when many companies were recruiting on college campuses, the recruiters would come to the campus and asked if a candidate could program. If they said yes, they became a programmer; and if they said no, they became a tester. The irony is that when the tester started work they had to use automated testing tools, and to use them they would have to know how to program. Developers encourage this view, many of whom consider testing not as rewarding or creative as programming. Consequently, many people would not intentionally or voluntarily seek a career in testing. However, I think they are mistaken. I ask my students which takes more creativity: creating the error or finding the error? Of course, the greatest creativity is needed to make excuses for the errors that developers created but testers didn’t find.

John: Where do you see testing tools technology heading?

Robin: I don't know that I see it heading anywhere, what I mean by that is that I would say that a year or two ago there was a lot of excitement about performance testing. I might be traveling in different circles, maybe organizations have gotten the issue of testing tools more under their belt. I haven’t been aware of the same level of excitement in the tool market as there was a few years ago. Back then, functional tools seemed to reach a plateau, and then most of the attention focused on performance testing; but it too now seems to have hit a plateau. I think that certainly there are areas of the web that continue to create and come up with new technologies and new test tool needs. There is greater attention on security now, and if you think about it security and performance is intimately related.

John: Here at Quotium Technologies we have been thinking a lot about the requirement of Independent Software Vendors (ISV) to demonstrate software scalability to their clients in the client’s environment. Can you recall any examples of ISV projects in your career where the ISV had to demonstrate scalability? What were the pitfalls and advice you'd have for ISVs in handling that issue?

Robin: Nothing comes to mind. A lot of my work is involved with presenting, training, and working directly with larger established companies and those with testing organizations. I think ISVs don’t fit those models and are not getting the training. Also, I participate in a lot of conferences, and I don’t see ISVs there. I suspect software companies are busy developing, not considering themselves as professional testing organizations.

John: Can you recall any scalability projects from the customer’s perspective?

Robin: For buyers in general, suitable use of testing is one of the hardest things for buyers to understand. For instance, very few buyers know how to structure an acquisition properly, and especially they fail to recognize how to use Proactive Testing. Yes, buyers need to define their acceptance criteria and test the ISV’s application; but its not just running tests after the software has been delivered. In my training and consulting, I also emphasize how getting a vendor to provide a plan for reasonable and suitable testing can be one of the buyer’s most valuable tools for managing the acquisition.

The effectiveness of both functional and performance testing depends primarily on having an effective testing process. Everyone says this, but for most folks it’s only words. A testing process defines what needs to be tested. An automated tool can’t do that; it only can assist the execution of the tests that have been identified. Proactive Testing as we present and practice it not only helps focus effort on the most important testing, but identifying reusable test designs can greatly enhance the speed and quantity of test automation; and most importantly, Proactive Testing helps developers speed delivery by preventing rework and showstopper errors.

John: Lastly, what are the trends in software development and testing?

Robin: It seems to me that once again things have quieted down, especially on the functional testing side. More and more organizations are less and less interested in writing automated scripts; they are looking for ways to automate the process. Therefore, more and more organizations are moving to data driven testing and the even more-efficient techniques using Action Words or similar frameworks, which involve some up-front technical effort to enable non technical people to create considerable numbers of automated tests. So far, though, these approaches seem mainly to be used for functional testing. However, while it's not very widely recognized, the same type of thing can apply to performance testing as well.

The other trend is related to these techniques for automating the automation; but it’s somewhat discouraging. To the best of my knowledge what still is not spoken about very much in organizations, regardless of what type of automated testing is conducted and whether home grown or whatever, is that the key element to effective test tool utilization continues to be the adequacy of the testing process. In fact, I fear that emphasis on the tools themselves tends to distract from identifying what to test, and especially doing it in a systematic and economic way that covers the risks and enables more reusable tests.

John: Thanks Robin for taking part in the Web Performance interview, I really appreciate you answers and talking to the readership and me.

Note: Robin and I continued to chat more and we discussed his recently released book, Discovering REAL Business Requirements for Software Project Success, within the context of the “bookends of software development,” that is, business process analysis at the start of a project and performance testing at the end. Robin’s book critiques and suggests important improvements to the conventional approaches used by many organizations in developing software. Robin points out that traditional development continues to encounter creep because developers continue to follow a “Field of Dreams” approach, thinking that whatever they decide to build must be what the business needs. Robin shows how to shift the focus to discovering the REAL, business requirements first—and using more than 21 ways to test that the business requirements are accurate and complete—before getting wrapped up in what is going to be built. While presented with respect to software, the same approach and techniques are equally valuable for quality, sales, and marketing.