Two types of tests

There are lots of diffefrent types of tests:

unit tests
functional tests
integration tests
system integration tests
ui tests
api tests
performance tests
load tests
acceptance tests
accessibility tests
security tests

But I like to group them into two primary categories:

Tests you can perform on the code itself, and tests that reqired a complete system to be deployed.

There are obviously exceptions, and it’s more of a spectrum, of what needs to be compiled, linked, combined, integrated, deployed, and what backend or front end pieces, as well as operating system or network connectivity or other external dependencies.

But most software nowadays requires some sort of deployment — to a server or device, and probably some sort of connection — to a file system, a database, or network service.

And much of it needs some sort of compilation, preprocessing, or integration with external dependencies. Even mcode written with interpreted languages like python or javascript will be written with dependence on common libraries.

While it is possible to test a single unit of code in isolation, it’s often not practical to exclude all external libraries (even if just to print or log output) and you usually need a runtime, libraries, or operating system to execute them on.

So I tend to not be too purist about unit tests vs integration tests — except to make the distinctions that my code — or the code that needs tested — should not be combined with other code that that needs tested (as part of your system) as opposed to code that you assume to be working as designed, although the quality of some external libraries is questionable, and you’re likely to encounter bugs or deficiencies in such libraries (open source or closed.)

The distinction I like to make is this:

Can the code (compiled or interpretted, linked or not) be tested before being deployed or not?

This is an important distinction, because it determines whether the test can be executed before or after said deployment — and that matters for where the test is executed in your delivery pipeline.

Another way this distinction is generally useful is — does the test matter to the developer or the end user? By which I mean not, who cares whether the test passes or fails, but who cares about the result of the test.

A developer test is a test that the developer who wrote the code is primarily interested in. The output of a developer test (whether unit or integration test) is useful to the developer of the code — the implementor — and not really to anyone else.

The assumption here is: Did the code do what I intended it to do?

This is in contrast to functional tests (to use another term which is perhaps too vague to be really useful) where the output is useful to the user (or designer) of the system under test.

That assumption is: Does the software actually do what I wanted it to do?

If the author of the code is both the designer and end user, these questions amount to the same thing, but much software is designed to be used by someone other than the developer, or rather the developer creates code designed by someone else, intended for others to use.

So the functionality of the system is actually being tested, instead of the correctness of the code itself.

To reiterate, my two general categories of testing are:

Developer tests (unit or integration) which are performed on code before it is deployed,

System tests (functional or end-to-end) which are performed on a system after it is deployed, which general have external dependencies on the file system, network, or other services (database, etc.)

That’s not to say that there aren’t more categories of tests, or that you can’t do functional testing with mocked external systems — or developer tests which require external dependencies. But that these two grouping are generally useful to think about for 3 reasons:

  1. Who cares about the test?
  2. Does the test have external dependencies that require additional setup?
  3. Can the test be performed before or after a deployment?

These two type of tests usually have other attributes:

The first group are small, fast, and simple
The second group are larger, slower, and more complex

This makes another advantage of developer tests that they can be executed frequently, and relied upon for repeatability, and are easy to debug (generally targeting a single object or function).

And to be fair, functional (or system) tests also have an advantage that they perform a real world scenario that development tests may not be able to anticipate, and they are concerned primarily with outcomes, not implementation.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s