1. Make sure software works
2. Make sure software does what it should
3. Make sure software doesn’t do what it shouldn’t do
4. Make sure software doesn’t break
   when you (the user) do something wrong, for example
   
   Bonus:
   
5. Make sure software delights the user

Most of the time, test automation is really only doing #1 — making sure that it works, by navigating around the application, and performing basic functions.

This is ok.

This is what automation is good at.

But you also need to do other things. Things that are harder. Things that are difficult for humans to figure out how to do, and even harder for computers, despite the hype around “AI”.

Sidebar: Artificial intelligence and artificial sweeteners

Artificial intelligence is like artificial sweetener. It tricks your brain without providing any of the benefits, like:

• Energy (in the form of calories)
• Pleasure (to the taste)
• Tooth decay

Artificial sweeteners only simulate the pleasure from taste, which is really an anticipation of energy. It’s like the dopamine from a drug, or a video game that tricks your brain into thinking you’ve accomplished something.
Likewise, AI only simulates thinking, and it’s outputs give you a false confidence that someone has thought about the implications.
A large language model (LLM) like ChatGPT literally has no idea what it has written, whether it is bad or good, right or wrong, self contradictory or repetitive, or if it makes any sense at all.
The generative AI models don’t know how many fingers a human should have, whether a picture is a cate or a dog, or the thing it draws is representational at all, much less if it is possible to exist in any real space or follows any rules of logic or consistency.
The idea of leaving testing up to generative computer “AI” models is preposterous, given that testing is supposed to answer exactly these types of questions.

1. Does it work?

As I said, making sure something works is the easy part. Does an application start when I launch it? Can I login? Can I see a report when I click a button?

But does it work right?

B. Does it do what it should?

This is the area of functional testing. How can I tell software does what it should unless I know what it should do?

For that you need requirements.

Some requirements can be inferred. A tester can often figure out if software that is working, is doing the right thing using common sense and their knowledge about a topic.

Sidebar: Shopping cats and usability dark patterns

Wouldn’t it be nice if your shopping cart at the grocery store could scan the items as you take them off the shelf and tell you how much you’re going to spend?They discovered that you’re less likely to buy as much stuff if you realize how much you’re spending.

Look for this feature in online shopping carts as competition vanishes. When it’s hard to figure out your total, you can be pretty sure we are in an effective monopoly.

But some requirements are subtle. Does this item calculate price per item or per weight? What taxes are applied to which people and what products?

And some requirements require specialized knowledge. Domain knowledge. Knowledge about the business developing the software, or knowledge about the how and what software will be used for. Medical diagnostics, or aeronautic controls for example.

Sidebar: Agile considered as a helix of combinatorial complexity

If you have the requirements, that is, you can perhaps test them — assuming you understand them. But in this day and age of big a “Agile” top down bureaucracy and time filling meaningless ceremonies and complex processes and un-user-friendly tools, requirements are less clear than ever.
But I digress. Again.

If you have tests (and automation) that is going to make sure software does what it should, you’re going to need to

1. Know what it should do, and

2. Map your tests to those requirements

That is, assuming your tests (which are software), are actually doing what they should do.

Oh yeah, and you also need to

3. Know how to verify that those requirements are being met.

Because it’s virtually impossible for anyone to understand, much less enumerate, ***all requirements***, it stands to reason, you won’t be able to automate them all, or track what they are doing.

Combine this with the many numerous ways you can accomplish something:

- click a button
- click back
- no click it again
- hit refresh
- and so on

And you have a nearly infinite variety of ways that requirements can be met.

Not to mention the potential number of ways software can break along the way.

Actually, I am going to mention it. Because that is our next point:

4. Will it break?

Using the software will tell you if it works, and more than likely, as a tester you will discover novel and interesting 

(some may say “perverse”, or “diabolical”, but we’re not developers or project managers here.)

ways the software can break.

In fact, good testers relish in it. They love finding bugs. They love breaking things. They love seeing the smoke come out and the server room catch on fire.

Cloud data centers have really made this less fun, but there are additional benefits (*ahem* risks) to running your code over the network to servers halfway across the world controlled by someone else. And additional ways things can go wrong.

And they (I should say “we”, because I am myself a tester) get even more satisfaction, the more esoteric or bizarre ways they can trigger these bugs.

Perhaps nothing gives us more delight than hearing a developer scratch there head and say “It worked for me!” when we can clearly prove that’s not the case in all cases. Or for all definitions of “work”.

Breaking things is the delight of the tester, and while there are tools that can put stress on software with high loads and random inputs, nothing beats a dumb human for making dumb mistakes.

And finally, since I like to do things out of order (to see if something breaks) we need to see what else software can do (that it probably shouldn’t do):

# X. Bugs

Have you ever come across an unexpected behavior in software and been told,

“That’s not a bug, that’s a feature”

No, dear developer, that’s a bug, not a feature.
If it’s doing what it’s not supposed to, it’s not supposed to do it.
So it stands to reason that any undocumented feature should be considered a bug.
But as we pointed out earlier, not every feature requirement can be documented, and the effort probably isn’t even worth it, because, let’s be honest: no one will ever read comprehensive documentation, much less test it all

• Every.
• Single.

• Time.
• Something changes.

What’s the difference between a bug and a defect?

Some would have you say spelling is the only difference. I disagree. I think the pronunciation is also different.

A defect is when something you wanted in the system isn’t in it. Something is missing. A requirement isn’t met.

A bug (as Admiral Grace Hopper *allegedly* found out the hard way) is when something you didn’t want gets into the system.

Whether it’s a moth or Richard Pryor doesn’t matter. The point is, it’s not supposed to be there. But it is.

Sometimes this breaks the system (like in Admiral Hopper’s case) other times, it doesn’t break the system (as in Richard Pryor’s case).

It could be a security issue, but it doesn’t have to be. It could just live there happily, taking up bits, and burning cycles and nobody ever notices anything is wrong (except whoever pays the AWS bill).

Anyway, it shouldn’t be there if it isn’t intended to be there, even if it’s beneficial. If you discover it, and it turns out useful, you can document it, and then it becomes a documented feature.

No, adding a note to the bug report “Working as intended” does not count as documenting a feature.

But, it’s very hard to prove a negative. That is that it doesn’t have a feature it shouldn’t have.

***

So to reiterate, there are 4 things that testing (or Quality Assurance) should be accomplishing:

1. Making sure it works

Automation, or any random user, can see that this is the case. However, just because something works, doesn’t mean that it does what it’s supposed to, that it doens’t do what it shouldn’t, that it will keep working when it comes into contact with the enemy — I mean users.

Smoke tests fit well into this category, but it should go beyond just making sure it doesn’t burst into flames when you first turn it on.

B. Making sure it does what it’s supposed to

You need to know what it’s supposed to do to test this. Some things are obvious, but in some cases, requirements are needed. But comprehensive documentation is not practical.

This is often considered functional testing. Some of this can be automated, but due to many factors (including the reasons above), it’s not practical to automate everything.

 4. Making sure it doesn’t break

This can be harder to prove. But it’s important. Just because something is working at one point, doesn’t mean it always will be.

Load & Stress testing are a part of this. But so is “monkey testing” or “chaos testing” which as the names imply, are unguided.

Testers with their pernicious creativity and reasoning abilities can go beyond random behavior and deliberately try to break things.

The goal here is to make the system stable.

X. Making sure it doesn’t do what it’s not supposed to do

This is the hardest part, but the funnest part of testing. Often when something breaks, (e.g. a buffer overrun), it can also have unexpected behavior.

It can have serious security implications, but also may cause usability issues.

Which brings us to our bonus point:

# Bonus: Making sure it delights the user.

Something can work flawlessly, be perfectly secure, fulfill all requirements, and still be an unmitigated pain in the neck to use.

In actuality, trying to make something robust, reliable, secure, and complete ***usually*** ends up harming usability.

Add to this the simple principle that someone who created the system is ***probably*** going to understand the system better than someone who didn’t, means that they may make assumptions about how to use it that are either not valid, or obvious to the intended user.

Usability testing is an important part of testing and pretty much can’t be automated (although I’d be interested to hear ideas about how you think it could.)

Usability testing is also often neglected, or not done from the user perspective.

Anyway, that’s all I have to say about that, for now.

Here is my answer to the question from Quora

What are some locator strategies that can be used in Selenium WebDriver?

Selenium WebDriver has several locator strategies — or methods for locating elements.

Whey you want to find an element, you need to locate it on the page. The way Selenium does this is by using Javascript to parse the HTML source content. In Javascript you can do the following:

document.getElementById(locator)
document.getElementsByName(locator)
document.getElementsByTagName(locator)
document.getElementsByClassName(locator)

WebDriver has corresponding locator strategies:

driver.findElement(By.id(locator))
driver.findElement(By.name(locator))
driver.findElement(By.tagName(locator))
driver.findElement(By.className(locator))

It also has additional methods for locating by XPATH, CSS Selector, and link text:

driver.findElement(By.xpath(locator))
driver.findElement(By.cssSelector(locator))
driver.findElement(By.linkText(locator))
driver.findElement(By.partialLinkText(locator))

XPath and CSS selectors are ways to parse the HTML document and give more precise locators including a combination of element tag hierarchies, attributes, CSS classes, and relative position (parent, child, sibling). I won’t go into details, but these are powerful strategies for parsing a document and finding specific elements based on several criteria.

LinkText and PartialLinkText searches for anchor <a> tags that contain the given text for the locator.

By.linkText(“Click Here”)
By.partialLinkText(“Click”)

WebDriver also has corresponding findElements() (plural) methods that can locate a list of matching elements. For instance, you can find all elements with tag name <div> or matching xpath //table/h1 (find all H1 tags within a table). By default, findElement() (singular) will return the first matching element.

Selenium 4 also introduced Relative Locators which can modify an existing locator with terms “above”, “below”, “rightOf”, “leftOf” or “near” (near meaning within 50 pixels). In practice, relative locators are often not reliable, because layout typically depends on a fixed screen size and layout. One use for relative locators is to check responsive layouts given a known screen size. For example, to make sure a button is below a div on mobile devices, but beside it on a full screen:

By mobileButton = RelativeLocator.with(By.id(“myButton”)).below(By.id(“myDiv”))
By desktopButton = RelativeLocator.with(By.id(“myButton”)).rightOf(By.id(“myDiv”))

Now, the next question is: Which locator strategy should I use — and why?

By.id is the most efficient locator, the most concise, and the least likely to change. But not every element has a unique id attribute. Use it when you can.

<button id=”login”>
driver.findElement(By.id(“login”))

By.name is useful for form elements, and is also concise and specific.

<input name=”email”>
driver.findElement(By.name(“email”))

Tag and and class name are often useful for finding all elements that match that specific criteria:

driver.findElements(By.tagName(“a”)) ← this will find all links on the page
driver.findElements(By.className(“dark”)) ← this will find all elements with the “dark” class attribute.

XPATH is definitely the most versatile, but can be very ugly and easy to break

driver.findElement(By.xpath(“//table/*/div[\@class=’result’]/a[contains(text(), ‘edit’)]") ← find the edit link in the first table that contains a div element with class name “result”

But CSS selectors can do most of the same things as XPATH (except finding parent, child, sibling, or text nodes) and is often more readable.

driver.findElement(By.cssSelector(“table div.result > a”)) ← find the first link in the first table within a div with class name “result”.

Note: CSS selector cannot find elements by specific text.

As you can see, CSS (and XPATH) locators can incorporate the above strategies (tag name, class name, id) into one locator. Many people prefer to use one of these locator strategies exclusively for consistency.

However, an important “strategy” when using XPATH or CSS selectors is to not use complex selectors that depend on the document hierarchy. You should try to find a unique locator as specifically as possible, by id, name, or tag/class combination that will not be likely to change as the page layout changes.

If you cannot identify a single element definitively, you can look for the closest unique parent element. Using a relative XPATH or CSS selector (different from a relative locator like “above” or “below”) from that parent is a good strategy.

driver.findElement(By.cssSelector(“#uniqueId > div”)) ← find the first div child of an element with the uniqueId.

In CSS Selectors:

div#uniqueID ← search for a <div id=”uniqueId”> element with ID attribute
div.className ← search for a <div class”=myClass”> element with class attribute

Personally, I recommend that given the choice between XPATH and CSS selectors, to choose CSS when possible. Both for readability, and as a practical consideration — web developers know CSS selectors well, but usually do not use XPATH.

Finally, you can locate one element, and then search for other elements below it by performing two searches.

driver.findElement(By.xpath(“//table”)).findElement(By.cssSelector(“.results”)) ← find the first table, then find the element with className “results”. 

This does incur a slight performance penalty by making multiple WebDriver findElement calls. You should try to find elements with a single locator when possibly, but not at the expense of either readability (complex locators) or maintainability (likely to change). These often coincide.

In summary, you should try to find unique locators for elements that will not break as the page layout changes. Finding elements by ID is the most efficient. XPATH, and then CSS selectors are the most versatile, and you can often get whatever you want with one of these two. You should strive for simple locators that identify an element uniquely, but avoid complex hierarchical locator strategies when possible because they can lead to difficult to maintain code.

Last week, I wrote about (one way) how to categorize tests.

And I promised a follow up about different ways to categorize tests. Or “how to slice the cake” as I put it.

https://fijiaaron.com/2023/02/09/how-do-you-categorize-tests/

This is not that follow up.

But after getting some feedback, I thought I’d expand a bit more on the “how” I categorizing tests.

So rather than talk about the different slices of cake, I’m going to talk about “why” I sliced it this way. If you get a smaller slice, it’s not (only) because I think you’re fat. And if you get a bigger pice, it’s not (just) because I like you more than your sister.

The title gives away the “how” — which was pointed out by a reader that I didn’t make very clear in my previous post. That is to say, I categories tests by “who cares about them”.

A developer test is obviously a test that a developer cares about. A user test is a test that a user cares about.

Maybe that’s why we let users do so much of our testing for us.

And so on.

In my current role, there are several gaps in “what” is being tested, and this leads to frustration with QA. The QA team, the QA process, the testers, the test automation framework, etc. It’s my job to help identify and solve this.

My first premise of testing is that you can’t test everything.
Which naturally leads to the second premise:
That there’s always room for improvement.

The question then, is where to improve? And that isn’t always obvious. Or different areas to improve may be obvious to different people. Because those people have different perspectives and different needs.

On a content focused website, it might sense that the content itself — and it’s presentation — are of primary importance. But it might be more or less important that the content can be found. Either by searching (or navigating) the site, or by search engine crawling. One perspective focuses on satisfying existing users of the site, the other perspective focusing on gaining new users. Which depends on your business model or stage.

But there are other concerns, and not just functional concerns. When you talk about non-functional concerns people talk about performance, or security, or usability. But what about operational stability? What about monitoring, error recovery, and risk.

One area I think that is mostly overlooked by QA are operational concerns.

How do the people who deploy and maintain and support your applications benefit from testing?

A quick answer is to have smoke tests.

By “smoke tests” I mean, in the technical electrical engineering sense of the term:

Does the smoke come out when you plug it in? [1]

• When you deploy a change, how quickly can you detect that you haven’t broken something —
  or done (significant) harm to the system?
• How can you tell that all parts of the system are working as expected?
• How can you verify that your change is doing what it was intended to do?
• Did anything else change and what was it impact?
• Are there any unanticipated regressions?
• Are all the parts of the system restored to normal capacity?
• What (if any) impact did it have on users?

Not all of these points are smoke tests. Where do you draw the line?

It doesn’t have to be a line between “smoke tests” and “full testing”. You can have gradual stages. There may be things that you can report on in:

• less than 10 seconds
• under 5 minutes
• about 1 hour
• at lease a full day
• several weeks or more

Again, these types of tests may be doing different things. And different people may care about them.

So I think testers should spend time thinking about the different people who might care about their tests, and plan accordingly. This also means budgeting time to work on tests that mean the most to the people who care about them. Because you can’t test everything.

A good exercise is to find out everyone who cares about the software product and find out what their concerns are. In QA, we often think about two groups of people:

1. The users (as represented by the product owner)
2. The developers (as represented by the developer sitting closest to you — or the one who gives you the most feedback).
   
   Not only should you ask them questions, but you should ask questions of other people including:
   
3. Sales – they might care more about the experience of new users,
4. Compliance & Legal – they might care more about accessibility, data retention, and privacy,
5. Operations – they might care more about performance under load, security, ability to roll back features, and making sure all parts of the system are communicating.

I’m sure there are lots more people who are about the system.

Conversation on this top sparked one discussion about a real world scenario that probably never would have occurred to a tester focusing on functional requirement from a user perspective who doesn’t understand the architecture — and doesn’t know the deployment process.

The way to address all these concerns in testing is to communicate. And I can’t think of a better way to communicate than to have cross functional teams where all perspectives are represented — and everyone cares about testing, not just the testers, and that testing isn’t something you do just before “throwing it over the wall” to production.

[1] Everyone knows that smoke is the magical substance that makes electronics work. Because when the smoke comes out, it stops working.

Starting a discussion on test types topics…This will probably become an outline for a presentation.

Not all tests types will be discussed and some are orthogonal, some not relevant (e.g. to QA). Most of all, definitions of test types are fuzzy and vary.

Starting with a simple list…no, starting with my take on categories. (There are 2 types of categories… )
I divide software testing (generally) into:

A. Developer tests and
B. Tester tests

There is a third type of tests also, which I’ll call non-tester tests. Things like A/B testing, usability testing, etc. which are performed with a different goal than verifying functionality or uncovering defects.

C. Other types of tests

There are also tests which might be verifying functionaly or finding defects, but have non-functional requirments, like security testing, performance testing, etc. that need special skills, or are not usually the focus of functional testers.

D. Non-functional requirements testing.

Developer tests include unit tests, integration tests, component tests, etc.There may be different decisions on what makes a “unit test”, whether it can access the file system, network, other classes, etc. (e.g. stict “London School”). But for my purpose I like to group them into tests that are performed by — and primarily “for” developers, and those performed by QA (and other) which are primarily designed with the product or user in mind.

I’m not talking about white-box vs black-box testing, though that category generally overlaps, meaning white-box tests are usually performed by developers. White/Black box has to do with whether you can see inside the system (the code, architecture, etc) or if you treat it as a “black box”, and are not concerned with the implementation. One advantage of white-box testing is that by knowing — or exploring — the architecture, you can better anticipate where to find bugs, but it’s also a case of missing the forest for the trees. Treating the system as a black box can mean focusing what the system “should” do, and not “how” it does it.

While the benefits are similar, and as I wrote the last paragraph, I was tempted to think maybe I do mean white-box and black-box. But not quite. Even if the people performing the tests are the same, and benefits correlate, the methods don’t. That is to say, my category doesn’t depend on whether it’s white- or black-box testing.

When I say developer tests, I mean tests that the developers care about (and nobody else really does), because they only benefit the developers — they help the developers write better code, faster. And so conversely, tester (maybe I should say user) tests primarly benefit the user, or the client. These tests are designed to help know that the product is functioning as desired, and to make sure that there are no (fewer) defects.

But another, more concrete way I break down tests into developer / tester tests (and “tester” doesn’t have to mean someone within a formal QA department) is to say that developer tests are performed on the code itself. And QA tests are perfomed on a delivered, whole product — or System.

In the case of web applications, this means tests that are performed before a deployment, and tests that are performed after a deployment. For desktop applications, that might be the compiled binary — and whatever else it needs to operate. Another way to look at that is to call these type of tests “System” tests.

So we have:

1. Developer tests and
2. System tests

However, when you say “System”, sometimes you think of the system architecture, and what you might be categorizing is how the system works — for the reason that we talked about above, vis white-box vs black-box. By understanding how the system works, you’re better able to test the system because you can then interact with it more efficiently:

• By targeting things that are more likely to break, and more likey to contain bugs. Areas where interfaces exist, complexity is hidden, etc.
• By interacting with those parts of the system directly (via API, DB query, etc) you can test system states and internal functionality that would be more difficult or slow (if not impossible) to expose from a user-focused test. We want to perform more of these tests because they are more likely to find subtle bugs in the system that are not likely to be exposed or anticipated by users, and because they are usually faster to execute (and less brittle) because they don’t depend on acting only as a user would. It’s also often the case that security exploits are found by accessing the system this way, by bypassing the user interface, or at least using it in a way it was not intended, to get more direct access to the system architecture. SQL injection and buffer overruns are exploits that are examples of using this type of attack vector.

With all that said, I’ve only defined the general categories of tests — by my definition of category. And I haven’t gotten down to definitions and strategy of different types of tests.

There are many ways to slice a cake, and that will be the next topic I tackle.

Following on my recent discovery of Selenium Manager, I checked out another project by Boni Garcia called Selenium-Jupiter.

It uses some advanced features of JUnit 5 (aka Junit-Jupiter) to make tests easier to write with less boilerplate code, part of that is using WebDriverManager to automatically instantiate WebDriver instances without downloading chromedriver, putting it in the path, setting webdriver.chrome.driver path in properties or as an environment variable, etc.

It starts with an JUnit 5 Extension — similar to JUnit 4 Runners – but you can have more than one extension.

@ExtendWith(SeleniumJupiter.class)
class MyTest { ... }

Then you can pass a WebDriver instance as a parameter to your test method (without having to create the instance — the Extension takes care of that for you.

@Test
void testWithChrome(ChromeDriver driver) { ... }

Or you can specify a different driver, for example FireFoxDriver, EdgeDriver, SafariDriver, etc.

@Test
void testWithFirefox(FirefoxDriver driver) { ... }

You can also specify whether to conditionally run a test with a specified driver, if docker containers are available, or if a site is up, and many other options available for WebDriverManager.

For instance, run this test only if you’re on a mac

@EnabledIfBrowserAvailable(SAFARI)
@ExtendWith(SeleniumJupiter.class)
class SafariTest {
    @Test
    void testWithSafari(SafariDriver driver) { ... }
}

Or run this test only if your Selenium server is up:

@EnabledIfDriverUrlOnline("http://localhost:4444/")
@ExtendWith(SeleniumJupiter.class)
class RemoteWebDriverTest {
     @Test
     void testRemote(@DriverUrl("http://localhost:4444/") { ... }
}

Or if Appium is running:

@EnabledIfDriverUrlOnline("http://localhost:4723/")
@ExtendWith(SeleniumJupiter.class)
class AppiumTest {
     @Test
     void testMobile(AppiumDriver driver) { ... }
}

So I was working with WebDriverManager this morning and one thing led to another, and I ended up browsing the Selenium source repo (as one does) and saw some curious commits (like these):

mark Selenium Manager implementations as beta

fix the framework conditionals for Selenium Manager

Add Selenium Manager support for Linux & Mac

from an old friend Titus Fortner.

I reached out to ask him about SeleniumManager — and it turns out it’s a replacement for WebDriverManager incorporated into the Selenium codebase (written by Boni Garcia, the original author of WebDriverManager, in Rust).

The various language bindings wrap a Rust binary (which for reasons I didn’t ask can’t be cross compiled with Selenium’s frankenstein 3rd or 4th generation custom build tool) so the SeleniumManager binary is packaged with the source.

Very cool, I thought, and then asked when it’s coming out.

Turns out, it was released today with Selenium 4.6.0

Here’s the official Selenium blog announcement:

Introducing Selenium Manager

Selenium 4.6.0 released