Starting a discussion on test types topics…This will probably become an outline for a presentation.

Not all tests types will be discussed and some are orthogonal, some not relevant (e.g. to QA). Most of all, definitions of test types are fuzzy and vary.

Starting with a simple list…no, starting with my take on categories. (There are 2 types of categories… )
I divide software testing (generally) into:

A. Developer tests and
B. Tester tests

There is a third type of tests also, which I’ll call non-tester tests. Things like A/B testing, usability testing, etc. which are performed with a different goal than verifying functionality or uncovering defects.

C. Other types of tests

There are also tests which might be verifying functionaly or finding defects, but have non-functional requirments, like security testing, performance testing, etc. that need special skills, or are not usually the focus of functional testers.

D. Non-functional requirements testing.

Developer tests include unit tests, integration tests, component tests, etc.There may be different decisions on what makes a “unit test”, whether it can access the file system, network, other classes, etc. (e.g. stict “London School”). But for my purpose I like to group them into tests that are performed by — and primarily “for” developers, and those performed by QA (and other) which are primarily designed with the product or user in mind.

I’m not talking about white-box vs black-box testing, though that category generally overlaps, meaning white-box tests are usually performed by developers. White/Black box has to do with whether you can see inside the system (the code, architecture, etc) or if you treat it as a “black box”, and are not concerned with the implementation. One advantage of white-box testing is that by knowing — or exploring — the architecture, you can better anticipate where to find bugs, but it’s also a case of missing the forest for the trees. Treating the system as a black box can mean focusing what the system “should” do, and not “how” it does it.

While the benefits are similar, and as I wrote the last paragraph, I was tempted to think maybe I do mean white-box and black-box. But not quite. Even if the people performing the tests are the same, and benefits correlate, the methods don’t. That is to say, my category doesn’t depend on whether it’s white- or black-box testing.

When I say developer tests, I mean tests that the developers care about (and nobody else really does), because they only benefit the developers — they help the developers write better code, faster. And so conversely, tester (maybe I should say user) tests primarly benefit the user, or the client. These tests are designed to help know that the product is functioning as desired, and to make sure that there are no (fewer) defects.

But another, more concrete way I break down tests into developer / tester tests (and “tester” doesn’t have to mean someone within a formal QA department) is to say that developer tests are performed on the code itself. And QA tests are perfomed on a delivered, whole product — or System.

In the case of web applications, this means tests that are performed before a deployment, and tests that are performed after a deployment. For desktop applications, that might be the compiled binary — and whatever else it needs to operate. Another way to look at that is to call these type of tests “System” tests.

So we have:

1. Developer tests and
2. System tests

However, when you say “System”, sometimes you think of the system architecture, and what you might be categorizing is how the system works — for the reason that we talked about above, vis white-box vs black-box. By understanding how the system works, you’re better able to test the system because you can then interact with it more efficiently:

• By targeting things that are more likely to break, and more likey to contain bugs. Areas where interfaces exist, complexity is hidden, etc.
• By interacting with those parts of the system directly (via API, DB query, etc) you can test system states and internal functionality that would be more difficult or slow (if not impossible) to expose from a user-focused test. We want to perform more of these tests because they are more likely to find subtle bugs in the system that are not likely to be exposed or anticipated by users, and because they are usually faster to execute (and less brittle) because they don’t depend on acting only as a user would. It’s also often the case that security exploits are found by accessing the system this way, by bypassing the user interface, or at least using it in a way it was not intended, to get more direct access to the system architecture. SQL injection and buffer overruns are exploits that are examples of using this type of attack vector.

With all that said, I’ve only defined the general categories of tests — by my definition of category. And I haven’t gotten down to definitions and strategy of different types of tests.

There are many ways to slice a cake, and that will be the next topic I tackle.