Starting a discussion on test types topics…This will probably become an outline for a presentation.

Not all tests types will be discussed and some are orthogonal, some not relevant (e.g. to QA). Most of all, definitions of test types are fuzzy and vary.

Starting with a simple list…no, starting with my take on categories. (There are 2 types of categories… )
I divide software testing (generally) into:

A. Developer tests and
B. Tester tests

There is a third type of tests also, which I’ll call non-tester tests. Things like A/B testing, usability testing, etc. which are performed with a different goal than verifying functionality or uncovering defects.

C. Other types of tests

There are also tests which might be verifying functionaly or finding defects, but have non-functional requirments, like security testing, performance testing, etc. that need special skills, or are not usually the focus of functional testers.

D. Non-functional requirements testing.

Developer tests include unit tests, integration tests, component tests, etc.There may be different decisions on what makes a “unit test”, whether it can access the file system, network, other classes, etc. (e.g. stict “London School”). But for my purpose I like to group them into tests that are performed by — and primarily “for” developers, and those performed by QA (and other) which are primarily designed with the product or user in mind.

I’m not talking about white-box vs black-box testing, though that category generally overlaps, meaning white-box tests are usually performed by developers. White/Black box has to do with whether you can see inside the system (the code, architecture, etc) or if you treat it as a “black box”, and are not concerned with the implementation. One advantage of white-box testing is that by knowing — or exploring — the architecture, you can better anticipate where to find bugs, but it’s also a case of missing the forest for the trees. Treating the system as a black box can mean focusing what the system “should” do, and not “how” it does it.

While the benefits are similar, and as I wrote the last paragraph, I was tempted to think maybe I do mean white-box and black-box. But not quite. Even if the people performing the tests are the same, and benefits correlate, the methods don’t. That is to say, my category doesn’t depend on whether it’s white- or black-box testing.

When I say developer tests, I mean tests that the developers care about (and nobody else really does), because they only benefit the developers — they help the developers write better code, faster. And so conversely, tester (maybe I should say user) tests primarly benefit the user, or the client. These tests are designed to help know that the product is functioning as desired, and to make sure that there are no (fewer) defects.

But another, more concrete way I break down tests into developer / tester tests (and “tester” doesn’t have to mean someone within a formal QA department) is to say that developer tests are performed on the code itself. And QA tests are perfomed on a delivered, whole product — or System.

In the case of web applications, this means tests that are performed before a deployment, and tests that are performed after a deployment. For desktop applications, that might be the compiled binary — and whatever else it needs to operate. Another way to look at that is to call these type of tests “System” tests.

So we have:

1. Developer tests and
2. System tests

However, when you say “System”, sometimes you think of the system architecture, and what you might be categorizing is how the system works — for the reason that we talked about above, vis white-box vs black-box. By understanding how the system works, you’re better able to test the system because you can then interact with it more efficiently:

• By targeting things that are more likely to break, and more likey to contain bugs. Areas where interfaces exist, complexity is hidden, etc.
• By interacting with those parts of the system directly (via API, DB query, etc) you can test system states and internal functionality that would be more difficult or slow (if not impossible) to expose from a user-focused test. We want to perform more of these tests because they are more likely to find subtle bugs in the system that are not likely to be exposed or anticipated by users, and because they are usually faster to execute (and less brittle) because they don’t depend on acting only as a user would. It’s also often the case that security exploits are found by accessing the system this way, by bypassing the user interface, or at least using it in a way it was not intended, to get more direct access to the system architecture. SQL injection and buffer overruns are exploits that are examples of using this type of attack vector.

With all that said, I’ve only defined the general categories of tests — by my definition of category. And I haven’t gotten down to definitions and strategy of different types of tests.

There are many ways to slice a cake, and that will be the next topic I tackle.

Following on my recent discovery of Selenium Manager, I checked out another project by Boni Garcia called Selenium-Jupiter.

It uses some advanced features of JUnit 5 (aka Junit-Jupiter) to make tests easier to write with less boilerplate code, part of that is using WebDriverManager to automatically instantiate WebDriver instances without downloading chromedriver, putting it in the path, setting webdriver.chrome.driver path in properties or as an environment variable, etc.

It starts with an JUnit 5 Extension — similar to JUnit 4 Runners – but you can have more than one extension.

@ExtendWith(SeleniumJupiter.class)
class MyTest { ... }

Then you can pass a WebDriver instance as a parameter to your test method (without having to create the instance — the Extension takes care of that for you.

@Test
void testWithChrome(ChromeDriver driver) { ... }

Or you can specify a different driver, for example FireFoxDriver, EdgeDriver, SafariDriver, etc.

@Test
void testWithFirefox(FirefoxDriver driver) { ... }

You can also specify whether to conditionally run a test with a specified driver, if docker containers are available, or if a site is up, and many other options available for WebDriverManager.

For instance, run this test only if you’re on a mac

@EnabledIfBrowserAvailable(SAFARI)
@ExtendWith(SeleniumJupiter.class)
class SafariTest {
    @Test
    void testWithSafari(SafariDriver driver) { ... }
}

Or run this test only if your Selenium server is up:

@EnabledIfDriverUrlOnline("http://localhost:4444/")
@ExtendWith(SeleniumJupiter.class)
class RemoteWebDriverTest {
     @Test
     void testRemote(@DriverUrl("http://localhost:4444/") { ... }
}

Or if Appium is running:

@EnabledIfDriverUrlOnline("http://localhost:4723/")
@ExtendWith(SeleniumJupiter.class)
class AppiumTest {
     @Test
     void testMobile(AppiumDriver driver) { ... }
}

So I was working with WebDriverManager this morning and one thing led to another, and I ended up browsing the Selenium source repo (as one does) and saw some curious commits (like these):

mark Selenium Manager implementations as beta

fix the framework conditionals for Selenium Manager

Add Selenium Manager support for Linux & Mac

from an old friend Titus Fortner.

I reached out to ask him about SeleniumManager — and it turns out it’s a replacement for WebDriverManager incorporated into the Selenium codebase (written by Boni Garcia, the original author of WebDriverManager, in Rust).

The various language bindings wrap a Rust binary (which for reasons I didn’t ask can’t be cross compiled with Selenium’s frankenstein 3rd or 4th generation custom build tool) so the SeleniumManager binary is packaged with the source.

Very cool, I thought, and then asked when it’s coming out.

Turns out, it was released today with Selenium 4.6.0

Here’s the official Selenium blog announcement:

Introducing Selenium Manager

Selenium 4.6.0 released

Greg Paskal on the “Craft of Testing” Youtube Channel, talks about the trap of “Going for Green” or writing tests with the aim of making sure they pass.

He has some great points and I recommend the video. Here are my comments from watching his post:

Two big differences I see with writing test automation vs traditional development:

1. Tests will need to be modified frequently — over a long time, not just until it “passes”.

2. Test failures cause friction, so you need to make sure that a failure means something, not just a pass.

What these two principles mean is that a test can’t just “work”. It needs to be able to let you know why it didn’t work — you can’t afford false positives because the cost is ignored tests — not just the failing test, but all others.

With a failing test, knowing why it failed and identifying the root cause (and production system that needs to be fixed to make the test pass) is only half the problem. When functionality, an interface, or some presupposition (data, environment, etc) changes, you need to be able to quickly adapt the test code to the new circumstance, and make sure that it not only works again — but that it is still performing the check intended.

That the test is still testing what you think it’s testing.

These challenges combine to make writing test automation code significantly different than writing most other code.

If you use VMWare vCloudDirector administration tool for managing your virtualization datacenter, you should be aware of the following vulnerability and patch your systems.

“An authenticated, high privileged malicious actor with network access to the VMware Cloud Director tenant or provider may be able to exploit a remote code execution vulnerability to gain access to the server,” VMware said in an advisory.

CVE-2022-22966 has a CVSS score of 9.1 out of 10.

Upgrading to version VMWARE Cloud Director version 10.1.4.1, 10.2.2.3 or 10.3.3 eliminates this vulnerability. The upgrade is hosted for download at kb.vmware.com.

If upgrading to a recommended version is not an option, you may apply this workaround  for CVE-2022-22966 in 9.7, 10.0, 10.1, 10.2 and 10.3

See more details at:

https://kb.vmware.com/s/article/88176

https://thehackernews.com/2022/04/critical-vmware-cloud-director-bug.html

“Are you only interested in test automation?”

I was asked this question casually, and here is my (detailed) response:

My opinion is that test automation serves 3 main purposes:

In each case, the automated tests’ primary roll is making sure the system works as expected and providing rapid, repetitive feedback.  

My opinion is that manual regression tests don’t necessarily make good automated regression tests — and that you can often develop them independently easier.

When I create automated tests, it is usually through the process of exploratory testing and analyzing requirements, and then determine when and whether to automate based on the following criteria:

Is this a check that will need to be done repeatedly?  

(Are the manual setup or verification steps difficult & slow to perform manually, or do they lend themselves easily to automation — e.g. database queries, system configuration, API calls, etc?)

Is this interface (UI or API) going to be stable or rapidly changing ?

(Will you need to frequently update the automation steps?)

Will this test provide enough value to pay the up front cost of developing automation?

(Is it harder to automated than to manually test? Or will it only need tested once?)

The reason I don’t recommend translating manual regression tests into automated tests is because not all these criteria can be met, and also that often, not all of the manual steps need to be reproduced in automation — or automation can do them more efficiently. 

For example: creating accounts via an API, verifying results via a SQL query, or bypassing UI navigation steps via deep linking, setting cookies, etc.

Automation also lends itself well to testing many data variations that are too tedious to perform manually, so you can end up testing some scenarios more thoroughly than you would with manual regression (due to time constraints).  

And sometimes, the cost of automating something is just not worth the effort, and it’s easier to perform manual checks.

All this said, I’m not opposed to doing requirements analysis and manual testing — in fact I consider it an important part of deciding what and how to automate. I just don’t think that automation should be considered a secondary step — because of my initial premise that the goal of automation is to accelerate those other rolls of development, testing, and delivery.

Yet another victory over MacOS!

To stop MacOS from rearranging your virtual desktops (after you have them just the way you want them):

Go to System Preferences > Mission Control
Uncheck “Automatically rearrange Spaces based on most recent use”

Thanks to:
https://www.appsntips.com/learn/how-to-stop-mac-spaces-from-rearranging-themselves-on-macos/

P.S. If virtual desktops (or “Spaces” as Apple calls them) are new to you:

Press “F3” to view your virtual desktops.

You can have multiple Spaces to group related windows and then swipe left & right between them using 3 fingers on your trackpad or magic mouse.

The popular web framework can be used to solve the problem of how to put text on a screen, but it’s almost like someone fed a machine the JavaScript specification and it iterated on a random sequence of code that parses until it hit on a result that creates an acceptable web page.

React’s syntax is bizarre, heavily uses obscure, little used language features that nobody knows about, and in fact only exist because the JavaScript spec is so loose.

There also appears to be no deliberate design or regard for conventions, readability, or structure. Even it’s English terminology looks like something generated by a machine that was fed a dictionary.

componentDidUpdate ???

With a little further reflection, I don’t think the AI accusation is that far off.

Really smart developers just out of college who are tasked with bizarre misguided questions like

How can we write PHP — in JavaScript?

Then, with no real world experience, and being hyperfocused on generating code come up with a solution that satisfies business who care nothing about code.

And then more really intelligent people learn to code by looking at that code and so on… and nobody stops to think about what should be done or how it can be done better. And code is never really maintained anymore — it’s easier to scrap it and just get funding to build a new company.

So a randomized algorithm that generates a solution optimized for a single task is created by a large group of people who think like computers comes up with an answer the same way it would answer the question —

Is this a picture of a cat?