Last week, I wrote about (one way) how to categorize tests.

And I promised a follow up about different ways to categorize tests. Or “how to slice the cake” as I put it.

https://fijiaaron.com/2023/02/09/how-do-you-categorize-tests/

This is not that follow up.

But after getting some feedback, I thought I’d expand a bit more on the “how” I categorizing tests.

So rather than talk about the different slices of cake, I’m going to talk about “why” I sliced it this way. If you get a smaller slice, it’s not (only) because I think you’re fat. And if you get a bigger pice, it’s not (just) because I like you more than your sister.

The title gives away the “how” — which was pointed out by a reader that I didn’t make very clear in my previous post. That is to say, I categories tests by “who cares about them”.

A developer test is obviously a test that a developer cares about. A user test is a test that a user cares about.

Maybe that’s why we let users do so much of our testing for us.

And so on.

In my current role, there are several gaps in “what” is being tested, and this leads to frustration with QA. The QA team, the QA process, the testers, the test automation framework, etc. It’s my job to help identify and solve this.

My first premise of testing is that you can’t test everything.
Which naturally leads to the second premise:
That there’s always room for improvement.

The question then, is where to improve? And that isn’t always obvious. Or different areas to improve may be obvious to different people. Because those people have different perspectives and different needs.

On a content focused website, it might sense that the content itself — and it’s presentation — are of primary importance. But it might be more or less important that the content can be found. Either by searching (or navigating) the site, or by search engine crawling. One perspective focuses on satisfying existing users of the site, the other perspective focusing on gaining new users. Which depends on your business model or stage.

But there are other concerns, and not just functional concerns. When you talk about non-functional concerns people talk about performance, or security, or usability. But what about operational stability? What about monitoring, error recovery, and risk.

One area I think that is mostly overlooked by QA are operational concerns.

How do the people who deploy and maintain and support your applications benefit from testing?

A quick answer is to have smoke tests.

By “smoke tests” I mean, in the technical electrical engineering sense of the term:

Does the smoke come out when you plug it in? [1]

• When you deploy a change, how quickly can you detect that you haven’t broken something —
  or done (significant) harm to the system?
• How can you tell that all parts of the system are working as expected?
• How can you verify that your change is doing what it was intended to do?
• Did anything else change and what was it impact?
• Are there any unanticipated regressions?
• Are all the parts of the system restored to normal capacity?
• What (if any) impact did it have on users?

Not all of these points are smoke tests. Where do you draw the line?

It doesn’t have to be a line between “smoke tests” and “full testing”. You can have gradual stages. There may be things that you can report on in:

• less than 10 seconds
• under 5 minutes
• about 1 hour
• at lease a full day
• several weeks or more

Again, these types of tests may be doing different things. And different people may care about them.

So I think testers should spend time thinking about the different people who might care about their tests, and plan accordingly. This also means budgeting time to work on tests that mean the most to the people who care about them. Because you can’t test everything.

A good exercise is to find out everyone who cares about the software product and find out what their concerns are. In QA, we often think about two groups of people:

1. The users (as represented by the product owner)
2. The developers (as represented by the developer sitting closest to you — or the one who gives you the most feedback).
   
   Not only should you ask them questions, but you should ask questions of other people including:
   
3. Sales – they might care more about the experience of new users,
4. Compliance & Legal – they might care more about accessibility, data retention, and privacy,
5. Operations – they might care more about performance under load, security, ability to roll back features, and making sure all parts of the system are communicating.

I’m sure there are lots more people who are about the system.

Conversation on this top sparked one discussion about a real world scenario that probably never would have occurred to a tester focusing on functional requirement from a user perspective who doesn’t understand the architecture — and doesn’t know the deployment process.

The way to address all these concerns in testing is to communicate. And I can’t think of a better way to communicate than to have cross functional teams where all perspectives are represented — and everyone cares about testing, not just the testers, and that testing isn’t something you do just before “throwing it over the wall” to production.

[1] Everyone knows that smoke is the magical substance that makes electronics work. Because when the smoke comes out, it stops working.

Starting a discussion on test types topics…This will probably become an outline for a presentation.

Not all tests types will be discussed and some are orthogonal, some not relevant (e.g. to QA). Most of all, definitions of test types are fuzzy and vary.

Starting with a simple list…no, starting with my take on categories. (There are 2 types of categories… )
I divide software testing (generally) into:

A. Developer tests and
B. Tester tests

There is a third type of tests also, which I’ll call non-tester tests. Things like A/B testing, usability testing, etc. which are performed with a different goal than verifying functionality or uncovering defects.

C. Other types of tests

There are also tests which might be verifying functionaly or finding defects, but have non-functional requirments, like security testing, performance testing, etc. that need special skills, or are not usually the focus of functional testers.

D. Non-functional requirements testing.

Developer tests include unit tests, integration tests, component tests, etc.There may be different decisions on what makes a “unit test”, whether it can access the file system, network, other classes, etc. (e.g. stict “London School”). But for my purpose I like to group them into tests that are performed by — and primarily “for” developers, and those performed by QA (and other) which are primarily designed with the product or user in mind.

I’m not talking about white-box vs black-box testing, though that category generally overlaps, meaning white-box tests are usually performed by developers. White/Black box has to do with whether you can see inside the system (the code, architecture, etc) or if you treat it as a “black box”, and are not concerned with the implementation. One advantage of white-box testing is that by knowing — or exploring — the architecture, you can better anticipate where to find bugs, but it’s also a case of missing the forest for the trees. Treating the system as a black box can mean focusing what the system “should” do, and not “how” it does it.

While the benefits are similar, and as I wrote the last paragraph, I was tempted to think maybe I do mean white-box and black-box. But not quite. Even if the people performing the tests are the same, and benefits correlate, the methods don’t. That is to say, my category doesn’t depend on whether it’s white- or black-box testing.

When I say developer tests, I mean tests that the developers care about (and nobody else really does), because they only benefit the developers — they help the developers write better code, faster. And so conversely, tester (maybe I should say user) tests primarly benefit the user, or the client. These tests are designed to help know that the product is functioning as desired, and to make sure that there are no (fewer) defects.

But another, more concrete way I break down tests into developer / tester tests (and “tester” doesn’t have to mean someone within a formal QA department) is to say that developer tests are performed on the code itself. And QA tests are perfomed on a delivered, whole product — or System.

In the case of web applications, this means tests that are performed before a deployment, and tests that are performed after a deployment. For desktop applications, that might be the compiled binary — and whatever else it needs to operate. Another way to look at that is to call these type of tests “System” tests.

So we have:

1. Developer tests and
2. System tests

However, when you say “System”, sometimes you think of the system architecture, and what you might be categorizing is how the system works — for the reason that we talked about above, vis white-box vs black-box. By understanding how the system works, you’re better able to test the system because you can then interact with it more efficiently:

• By targeting things that are more likely to break, and more likey to contain bugs. Areas where interfaces exist, complexity is hidden, etc.
• By interacting with those parts of the system directly (via API, DB query, etc) you can test system states and internal functionality that would be more difficult or slow (if not impossible) to expose from a user-focused test. We want to perform more of these tests because they are more likely to find subtle bugs in the system that are not likely to be exposed or anticipated by users, and because they are usually faster to execute (and less brittle) because they don’t depend on acting only as a user would. It’s also often the case that security exploits are found by accessing the system this way, by bypassing the user interface, or at least using it in a way it was not intended, to get more direct access to the system architecture. SQL injection and buffer overruns are exploits that are examples of using this type of attack vector.

With all that said, I’ve only defined the general categories of tests — by my definition of category. And I haven’t gotten down to definitions and strategy of different types of tests.

There are many ways to slice a cake, and that will be the next topic I tackle.

Following on my recent discovery of Selenium Manager, I checked out another project by Boni Garcia called Selenium-Jupiter.

It uses some advanced features of JUnit 5 (aka Junit-Jupiter) to make tests easier to write with less boilerplate code, part of that is using WebDriverManager to automatically instantiate WebDriver instances without downloading chromedriver, putting it in the path, setting webdriver.chrome.driver path in properties or as an environment variable, etc.

It starts with an JUnit 5 Extension — similar to JUnit 4 Runners – but you can have more than one extension.

@ExtendWith(SeleniumJupiter.class)
class MyTest { ... }

Then you can pass a WebDriver instance as a parameter to your test method (without having to create the instance — the Extension takes care of that for you.

@Test
void testWithChrome(ChromeDriver driver) { ... }

Or you can specify a different driver, for example FireFoxDriver, EdgeDriver, SafariDriver, etc.

@Test
void testWithFirefox(FirefoxDriver driver) { ... }

You can also specify whether to conditionally run a test with a specified driver, if docker containers are available, or if a site is up, and many other options available for WebDriverManager.

For instance, run this test only if you’re on a mac

@EnabledIfBrowserAvailable(SAFARI)
@ExtendWith(SeleniumJupiter.class)
class SafariTest {
    @Test
    void testWithSafari(SafariDriver driver) { ... }
}

Or run this test only if your Selenium server is up:

@EnabledIfDriverUrlOnline("http://localhost:4444/")
@ExtendWith(SeleniumJupiter.class)
class RemoteWebDriverTest {
     @Test
     void testRemote(@DriverUrl("http://localhost:4444/") { ... }
}

Or if Appium is running:

@EnabledIfDriverUrlOnline("http://localhost:4723/")
@ExtendWith(SeleniumJupiter.class)
class AppiumTest {
     @Test
     void testMobile(AppiumDriver driver) { ... }
}

So I was working with WebDriverManager this morning and one thing led to another, and I ended up browsing the Selenium source repo (as one does) and saw some curious commits (like these):

mark Selenium Manager implementations as beta

fix the framework conditionals for Selenium Manager

Add Selenium Manager support for Linux & Mac

from an old friend Titus Fortner.

I reached out to ask him about SeleniumManager — and it turns out it’s a replacement for WebDriverManager incorporated into the Selenium codebase (written by Boni Garcia, the original author of WebDriverManager, in Rust).

The various language bindings wrap a Rust binary (which for reasons I didn’t ask can’t be cross compiled with Selenium’s frankenstein 3rd or 4th generation custom build tool) so the SeleniumManager binary is packaged with the source.

Very cool, I thought, and then asked when it’s coming out.

Turns out, it was released today with Selenium 4.6.0

Here’s the official Selenium blog announcement:

Introducing Selenium Manager

Selenium 4.6.0 released

Greg Paskal on the “Craft of Testing” Youtube Channel, talks about the trap of “Going for Green” or writing tests with the aim of making sure they pass.

He has some great points and I recommend the video. Here are my comments from watching his post:

Two big differences I see with writing test automation vs traditional development:

1. Tests will need to be modified frequently — over a long time, not just until it “passes”.

2. Test failures cause friction, so you need to make sure that a failure means something, not just a pass.

What these two principles mean is that a test can’t just “work”. It needs to be able to let you know why it didn’t work — you can’t afford false positives because the cost is ignored tests — not just the failing test, but all others.

With a failing test, knowing why it failed and identifying the root cause (and production system that needs to be fixed to make the test pass) is only half the problem. When functionality, an interface, or some presupposition (data, environment, etc) changes, you need to be able to quickly adapt the test code to the new circumstance, and make sure that it not only works again — but that it is still performing the check intended.

That the test is still testing what you think it’s testing.

These challenges combine to make writing test automation code significantly different than writing most other code.

If you use VMWare vCloudDirector administration tool for managing your virtualization datacenter, you should be aware of the following vulnerability and patch your systems.

“An authenticated, high privileged malicious actor with network access to the VMware Cloud Director tenant or provider may be able to exploit a remote code execution vulnerability to gain access to the server,” VMware said in an advisory.

CVE-2022-22966 has a CVSS score of 9.1 out of 10.

Upgrading to version VMWARE Cloud Director version 10.1.4.1, 10.2.2.3 or 10.3.3 eliminates this vulnerability. The upgrade is hosted for download at kb.vmware.com.

If upgrading to a recommended version is not an option, you may apply this workaround  for CVE-2022-22966 in 9.7, 10.0, 10.1, 10.2 and 10.3

See more details at:

https://kb.vmware.com/s/article/88176

https://thehackernews.com/2022/04/critical-vmware-cloud-director-bug.html