Web Performance: The Interview with Jim Kandler

June 2004

Sun	Mon	Tue	Wed	Thu	Fri	Sat
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30

I met Jim Kandler at the 2004 StarEast Show in Orlando a few weeks ago. We sat down for lunch and Jim told me all about his new defect detection model. Jim is still defining his model and he needs help from the industry. Read this interview to learn more. Jim runs a consultancy called Medical Software.

John: Jim, please tell us something about yourself, your background?

Jim: I've been an engineer all my life, got into testing and software quality because I was always breaking software. Colleagues would come to me, tell me they thought they had their application working, and I would break the application within 5 minutes. That's why I got into this crazy field. I've specialized in medical products; I got into it because the field really benefits a lot of people, an altruistic sort of thing. I now run a freelance consulting business.

John: When we spoke at StarEast you told me that you were building a model to determine where testing time should be concentrated in the software development process to identify the most bugs. I understand you started to develop this model because you wondered if the many testing myths about testing were actually true. What have you discovered in your research so far?

Jim: I learned of a few things that happen, the model shows you where defects get injected. It becomes obvious that the bugs are getting injected at the beginning of a project, the bugs are then being found in testing. The faults were all injected at the requirements and coding stages. The model gives you this different view. The defects are not injected at the testing stage.

John: So you are saying that there was a myth that defects were injected in the testing phase.

Jim: Yes, people think defects are injected at the testing stage. If a requirement were described incorrectly, the requirement would be wrong. Design and coding were then enacted with poor requirements.

People can do testing on requirements, that's what some of the test driven design is trying to do. Test-driven design is a way of prototyping your requirements, whenever they build out a prototype they are actually proving their models. They could bring a testing person to provide some more rigorous testing of those models.

Again, regarding my model, I found some modeling of defect detection rates. AT&T did some work years ago, through software requirements engineering. They found when they built these models, your defection rate reduces over time, it becomes harder and harder to find defects, the efficiency drops as you do more testing. The greater the level of testing the more expensive it becomes to do testing. I might spend 2 hours to remove 10 bugs, another 4 hours to find another 10 bugs, 6 hours for the next 10 bugs. The cost goes up.

I found at the 35% bug detection level, the costs of bug removal become quite high. So intuitively, go back to your defect detection model, test throughout the process, and keep to 35%. Inexperienced development practitioners wonder why they cannot get out of the system- testing phase.

There are always more defects than you remove. Every application has defects; you cannot get rid of all of the defects. I am trying to refine the model, and at some point, I will want to try and run the model against several different companies (Contact Jim here if you want to get involved.)

John: What has been your experience with companies adding additional functionality to commercial automated testing tools?

Jim: No first hand experience, I have not used any one tool long enough. I noticed a lot of them have poor reliability. On some projects we send back testing tools, the tool was un-reliable. This still tends to be a true, many of testing tools are more poorly built than the applications you are trying to use them against. We ended up finding the tool was unusable for us, we thought we were testing the application, but there was something wrong with the tool. If anyone has expectations for quality, it's a QA person. A QA person would be very upset if they have to use an unreliable tool.

John: What functionality would you look for in a good automated performance-testing tool?

Jim: I think most have the functionality that is needed for today's testing environment; to be able to do table driven testing, take some raw data, drive a table driven scenarios. Many of testing tools do secure protocols, most of the big ones allow you to do some recording and embed that in your scripts for web performance testing. The tools need to be doing some error handling, informing the user of errors, if the pages are not being returned the users needs to know that.

The other good tools allow you to coordinate multiple engines, so you can launch your scripts against multiple engines.

I did find a nice feature on one load test tool recently; the tool had a distribution function, allowing you to proportionally distribute the functionality based on the proportions you want. For example, you may want to have 20% of your users, conduct a search, 20% do other things, and 60% of users log in and sit at the home page. With three different scenarios, you can set up a distribution % per function.

John: Where do you see testing tools technology heading?

Jim: Testing tools have been a silo application; there are many silos in software development, and they hold all kinds of data. A number of the testing tool companies are starting to interconnect their tools, Mercury has Testdirector, allowing you to manage test cases, link to requirements, actually integrates the tools across the process, keeps you from generating unique silos that don't allow you to communicate.

I see a lot of people who are upset with buying tools that are just silos, at Stareast we saw a lot of this going on; we saw this across applications that communicate with one another.

John: Lastly, what are the trends in software development and testing?

Jim: People are trying to test more in the requirements phase, taking a look at rapid application development. A continuous review of deliverables is the process they use for development. Lint and syntax checkers are common. People are now starting to use context checkers, similar to spell checkers and grammar checkers. Context checkers are taking this testing up a notch, something that is cost effective.

John: Thanks Jim for some great answers to my questions.

Web Performance

Quotium Technologies blog all about testing web applications

Recent Posts

June 2004

Blogs

Recent Comments

The Interview with Jim Kandler

Comments

About

Archives

Categories

Testing Tools