QA testing should do these 4 things:

  1. Make sure software works
  2. Make sure software does what it should
  3. Make sure software doesn’t do what it shouldn’t do
  4. Make sure software doesn’t break
    when you (the user) do something wrong, for example

  5. Make sure software delights the user

Most of the time, test automation is really only doing #1 — making sure that it works, by navigating around the application, and performing basic functions.

This is ok.

This is what automation is good at.

But you also need to do other things. Things that are harder. Things that are difficult for humans to figure out how to do, and even harder for computers, despite the hype around “AI”.

Sidebar: Artificial intelligence and artificial sweeteners

Artificial intelligence is like artificial sweetener. It tricks your brain without providing any of the benefits, like:

  • Energy (in the form of calories)
  • Pleasure (to the taste)
  • Tooth decay

Artificial sweeteners only simulate the pleasure from taste, which is really an anticipation of energy. It’s like the dopamine from a drug, or a video game that tricks your brain into thinking you’ve accomplished something.
Likewise, AI only simulates thinking, and it’s outputs give you a false confidence that someone has thought about the implications.
A large language model (LLM) like ChatGPT literally has no idea what it has written, whether it is bad or good, right or wrong, self contradictory or repetitive, or if it makes any sense at all.
The generative AI models don’t know how many fingers a human should have, whether a picture is a cate or a dog, or the thing it draws is representational at all, much less if it is possible to exist in any real space or follows any rules of logic or consistency.
The idea of leaving testing up to generative computer “AI” models is preposterous, given that testing is supposed to answer exactly these types of questions.

1. Does it work?

As I said, making sure something works is the easy part. Does an application start when I launch it? Can I login? Can I see a report when I click a button?

But does it work right?

B. Does it do what it should?

This is the area of functional testing. How can I tell software does what it should unless I know what it should do?

For that you need requirements.

Some requirements can be inferred. A tester can often figure out if software that is working, is doing the right thing using common sense and their knowledge about a topic.

Sidebar: Shopping cats and usability dark patterns

Wouldn’t it be nice if your shopping cart at the grocery store could scan the items as you take them off the shelf and tell you how much you’re going to spend?They discovered that you’re less likely to buy as much stuff if you realize how much you’re spending.

Look for this feature in online shopping carts as competition vanishes. When it’s hard to figure out your total, you can be pretty sure we are in an effective monopoly.

But some requirements are subtle. Does this item calculate price per item or per weight? What taxes are applied to which people and what products?

And some requirements require specialized knowledge. Domain knowledge. Knowledge about the business developing the software, or knowledge about the how and what software will be used for. Medical diagnostics, or aeronautic controls for example.

Sidebar: Agile considered as a helix of combinatorial complexity

If you have the requirements, that is, you can perhaps test them — assuming you understand them. But in this day and age of big a “Agile” top down bureaucracy and time filling meaningless ceremonies and complex processes and un-user-friendly tools, requirements are less clear than ever.
But I digress. Again.

If you have tests (and automation) that is going to make sure software does what it should, you’re going to need to

1. Know what it should do, and

2. Map your tests to those requirements

That is, assuming your tests (which are software), are actually doing what they should do.

Oh yeah, and you also need to

3. Know how to verify that those requirements are being met.

Because it’s virtually impossible for anyone to understand, much less enumerate, ***all requirements***, it stands to reason, you won’t be able to automate them all, or track what they are doing.

Combine this with the many numerous ways you can accomplish something:

- click a button
- click back
- no click it again
- hit refresh
- and so on

And you have a nearly infinite variety of ways that requirements can be met.

Not to mention the potential number of ways software can break along the way.

Actually, I am going to mention it. Because that is our next point:

4. Will it break?

Using the software will tell you if it works, and more than likely, as a tester you will discover novel and interesting 

(some may say “perverse”, or “diabolical”, but we’re not developers or project managers here.)

ways the software can break.

In fact, good testers relish in it. They love finding bugs. They love breaking things. They love seeing the smoke come out and the server room catch on fire.

Cloud data centers have really made this less fun, but there are additional benefits (*ahem* risks) to running your code over the network to servers halfway across the world controlled by someone else. And additional ways things can go wrong.

And they (I should say “we”, because I am myself a tester) get even more satisfaction, the more esoteric or bizarre ways they can trigger these bugs.

Perhaps nothing gives us more delight than hearing a developer scratch there head and say “It worked for me!” when we can clearly prove that’s not the case in all cases. Or for all definitions of “work”.

Breaking things is the delight of the tester, and while there are tools that can put stress on software with high loads and random inputs, nothing beats a dumb human for making dumb mistakes.

And finally, since I like to do things out of order (to see if something breaks) we need to see what else software can do (that it probably shouldn’t do):

# X. Bugs

Have you ever come across an unexpected behavior in software and been told,

“That’s not a bug, that’s a feature”

No, dear developer, that’s a bug, not a feature.
If it’s doing what it’s not supposed to, it’s not supposed to do it.
So it stands to reason that any undocumented feature should be considered a bug.
But as we pointed out earlier, not every feature requirement can be documented, and the effort probably isn’t even worth it, because, let’s be honest: no one will ever read comprehensive documentation, much less test it all

  • Every.
  • Single.
  • Time.
  • Something changes.

What’s the difference between a bug and a defect?

Some would have you say spelling is the only difference. I disagree. I think the pronunciation is also different.

A defect is when something you wanted in the system isn’t in it. Something is missing. A requirement isn’t met.

A bug (as Admiral Grace Hopper *allegedly* found out the hard way) is when something you didn’t want gets into the system.

Whether it’s a moth or Richard Pryor doesn’t matter. The point is, it’s not supposed to be there. But it is.

Sometimes this breaks the system (like in Admiral Hopper’s case) other times, it doesn’t break the system (as in Richard Pryor’s case).

It could be a security issue, but it doesn’t have to be. It could just live there happily, taking up bits, and burning cycles and nobody ever notices anything is wrong (except whoever pays the AWS bill).

Anyway, it shouldn’t be there if it isn’t intended to be there, even if it’s beneficial. If you discover it, and it turns out useful, you can document it, and then it becomes a documented feature.

No, adding a note to the bug report “Working as intended” does not count as documenting a feature.

But, it’s very hard to prove a negative. That is that it doesn’t have a feature it shouldn’t have.


So to reiterate, there are 4 things that testing (or Quality Assurance) should be accomplishing:

1. Making sure it works

Automation, or any random user, can see that this is the case. However, just because something works, doesn’t mean that it does what it’s supposed to, that it doens’t do what it shouldn’t, that it will keep working when it comes into contact with the enemy — I mean users.

Smoke tests fit well into this category, but it should go beyond just making sure it doesn’t burst into flames when you first turn it on.

B. Making sure it does what it’s supposed to

You need to know what it’s supposed to do to test this. Some things are obvious, but in some cases, requirements are needed. But comprehensive documentation is not practical.

This is often considered functional testing. Some of this can be automated, but due to many factors (including the reasons above), it’s not practical to automate everything.

 4. Making sure it doesn’t break

This can be harder to prove. But it’s important. Just because something is working at one point, doesn’t mean it always will be.

Load & Stress testing are a part of this. But so is “monkey testing” or “chaos testing” which as the names imply, are unguided.

Testers with their pernicious creativity and reasoning abilities can go beyond random behavior and deliberately try to break things.

The goal here is to make the system stable.

X. Making sure it doesn’t do what it’s not supposed to do

This is the hardest part, but the funnest part of testing. Often when something breaks, (e.g. a buffer overrun), it can also have unexpected behavior.

It can have serious security implications, but also may cause usability issues.

Which brings us to our bonus point:

# Bonus: Making sure it delights the user.

Something can work flawlessly, be perfectly secure, fulfill all requirements, and still be an unmitigated pain in the neck to use.

In actuality, trying to make something robust, reliable, secure, and complete ***usually*** ends up harming usability.

Add to this the simple principle that someone who created the system is ***probably*** going to understand the system better than someone who didn’t, means that they may make assumptions about how to use it that are either not valid, or obvious to the intended user.

Usability testing is an important part of testing and pretty much can’t be automated (although I’d be interested to hear ideas about how you think it could.)

Usability testing is also often neglected, or not done from the user perspective.

Anyway, that’s all I have to say about that, for now.

What are some Selenium WebDriver locator strategies?

Here is my answer to the question from Quora

What are some locator strategies that can be used in Selenium WebDriver?

Selenium WebDriver has several locator strategies — or methods for locating elements.

Whey you want to find an element, you need to locate it on the page. The way Selenium does this is by using Javascript to parse the HTML source content. In Javascript you can do the following:


WebDriver has corresponding locator strategies:


It also has additional methods for locating by XPATH, CSS Selector, and link text:


XPath and CSS selectors are ways to parse the HTML document and give more precise locators including a combination of element tag hierarchies, attributes, CSS classes, and relative position (parent, child, sibling). I won’t go into details, but these are powerful strategies for parsing a document and finding specific elements based on several criteria.

LinkText and PartialLinkText searches for anchor <a> tags that contain the given text for the locator.

By.linkText(“Click Here”)

WebDriver also has corresponding findElements() (plural) methods that can locate a list of matching elements. For instance, you can find all elements with tag name <div> or matching xpath //table/h1 (find all H1 tags within a table). By default, findElement() (singular) will return the first matching element.

Selenium 4 also introduced Relative Locators which can modify an existing locator with terms “above”, “below”, “rightOf”, “leftOf” or “near” (near meaning within 50 pixels). In practice, relative locators are often not reliable, because layout typically depends on a fixed screen size and layout. One use for relative locators is to check responsive layouts given a known screen size. For example, to make sure a button is below a div on mobile devices, but beside it on a full screen:

By mobileButton = RelativeLocator.with(“myButton”)).below(“myDiv”))
By desktopButton = RelativeLocator.with(“myButton”)).rightOf(“myDiv”))

Now, the next question is: Which locator strategy should I use — and why? is the most efficient locator, the most concise, and the least likely to change. But not every element has a unique id attribute. Use it when you can.

<button id=”login”>
driver.findElement(“login”)) is useful for form elements, and is also concise and specific.

<input name=”email”>

Tag and and class name are often useful for finding all elements that match that specific criteria:

driver.findElements(By.tagName(“a”)) ← this will find all links on the page
driver.findElements(By.className(“dark”)) ← this will find all elements with the “dark” class attribute.

XPATH is definitely the most versatile, but can be very ugly and easy to break

driver.findElement(By.xpath(“//table/*/div[\@class=’result’]/a[contains(text(), ‘edit’)]") ← find the edit link in the first table that contains a div element with class name “result”

But CSS selectors can do most of the same things as XPATH (except finding parent, child, sibling, or text nodes) and is often more readable.

driver.findElement(By.cssSelector(“table div.result > a”)) ← find the first link in the first table within a div with class name “result”.

Note: CSS selector cannot find elements by specific text.

As you can see, CSS (and XPATH) locators can incorporate the above strategies (tag name, class name, id) into one locator. Many people prefer to use one of these locator strategies exclusively for consistency.

However, an important “strategy” when using XPATH or CSS selectors is to not use complex selectors that depend on the document hierarchy. You should try to find a unique locator as specifically as possible, by id, name, or tag/class combination that will not be likely to change as the page layout changes.

If you cannot identify a single element definitively, you can look for the closest unique parent element. Using a relative XPATH or CSS selector (different from a relative locator like “above” or “below”) from that parent is a good strategy.

driver.findElement(By.cssSelector(“#uniqueId > div”)) ← find the first div child of an element with the uniqueId.

In CSS Selectors:

div#uniqueID ← search for a <div id=”uniqueId”> element with ID attribute
div.className ← search for a <div class”=myClass”> element with class attribute

Personally, I recommend that given the choice between XPATH and CSS selectors, to choose CSS when possible. Both for readability, and as a practical consideration — web developers know CSS selectors well, but usually do not use XPATH.

Finally, you can locate one element, and then search for other elements below it by performing two searches.

driver.findElement(By.xpath(“//table”)).findElement(By.cssSelector(“.results”)) ← find the first table, then find the element with className “results”. 

This does incur a slight performance penalty by making multiple WebDriver findElement calls. You should try to find elements with a single locator when possibly, but not at the expense of either readability (complex locators) or maintainability (likely to change). These often coincide.

In summary, you should try to find unique locators for elements that will not break as the page layout changes. Finding elements by ID is the most efficient. XPATH, and then CSS selectors are the most versatile, and you can often get whatever you want with one of these two. You should strive for simple locators that identify an element uniquely, but avoid complex hierarchical locator strategies when possible because they can lead to difficult to maintain code.

How to get started writing unit tests on a large complex codebase without existing tests

A little bit at a time.

Developing unit tests for a large scale application can be challenging. Especially if it was written without any tests, or with testing in mind.

Writing unit tests requires code to be modular — so you can test a unit in isolation. One of the biggest benefits of writing unit tests during development is not that your code will be well tested, but that it helps you think about writing it in a way that different parts of the application (units) can be tested without interacting with the whole application.

But if it hasn’t been written that way, it may require significant refactoring to be testable. Or it may require mocking, or creating fake services, or injected databases with dummy data. And this can make tests complex, brittle, and unreliable.

So a good way to start unit testing on a code base that is resistant to testing in isolation because it is complex & tightly coupled, is to add unit tests slowly, as you modify the codebase.

Find an area that is complex (and ideally has a bug that needs to be fixed). Once you identify the piece of code that you need to test (by looking at log files, stepping through it with a debugger, or adding “print” statements), work on recreating that manual verification process in a unit test.

Write a test that describes behavior you want to test (or defect you want to expose.)

Unit tests should not depend on a deployed application, external services, or need you to create many mocks or stubs. You do not want to do this, for the reasons mentioned above.

So if you can find a piece of logic, isolate it, and refactor the code so that you can test this in isolation.

You may need to mock 1 or 2 external interfaces, but don’t go overboard. Too many dependencies is a sign that your code is too tightly couples. If you find this is the case, you may want to start with unit tests in an area that is simpler to isolate. You can tackle the more complex areas of your codebase later — when you have more tests written, and refactoring it becomes less risky.

Another thing you do not want to do is create a bunch of simple unit tests for easy methods that have little or no logic — getters & setters, constructors, helper methods that just hide ugly APIs with verbose setup steps, etc. If you have code that is very unlikely to ever change or break, don’t write unit tests for it. Every test you write adds to your codebase, which means adding to maintenance cost. Don’t write unit tests just to increase numbers or code coverage. Make every test count.

One last thing to avoid is writing unit test that describe functionality or depend on the implementation. Tests with a lot of mocks or data setup or several assertions are probably because of this. By writing tests to the implementation, you may be verifying that it works as intended, but it will make it hard to refactor the codebase in the future. A lot of projects abandon unit testing when they find that the existing tests cause friction in refactoring, or worse, tests that end up getting ignored because they are not keeping up to date.

So to reiterate, the answer to getting started writing unit tests on a large existing codebase is to start slowly, adding tests as you work on specific areas of the codebase, refactoring the code as you work to make testing in isolation easier, and avoiding (at first) areas that are too difficult to break down into unit tests, and areas that are too simple to ever fail. Make ever test count, and make it tell you something useful, and don’t set up code coverage goals or write more tests just to increase numbers.

Here is the question on Quora that this answer addresses:

What is Selenium? Does it support multiple users?

Selenium is a programming tool used to automate the browser — simulating a real user by opening windows, going to URLs, clicking buttons, filling in forms, etc. Selenium is primarily used for test automation, but can also be used for other things.

Selenium is available as a library for most popular programming languages, including Java, JavaScript, TypeScript, C#, Python, Ruby, PHP, etc.

It is also referred to as Selenium WebDriver, because there were two different projects (Selenium & WebDriver) which did similar things and eventually merged. Selenium uses the WebDriver protocol (a W3C standard now) to communicate over the network via a REST API. This allows for remote automation,

There are other tools associated with Selenium, including Selenium Grid — which enables remote execution of automation and Selenium IDE — which allows you to record and play back automated steps without writing code, and has (limited) ability to to export from Selenium IDE to code that can run independent.

Selenium IDE does not support multiple users, but scripts can be exported and shared from one user to another.

Selenium Grid allows for parallel remote execution of Selenium scripts, which allows multiple people (or tests) to execute tests at the same time.

The concept of “supporting multiple users” does not really make sense in terms of Selenium as an open source development tool or coding library.

It would be like saying:
“Does Microsoft Word support multiple users?” or
“Does the Java programming language support multiple users?”

In the case of Microsoft Word, every user that has the program can use it, but collaboration is (primarily) done outside of the tool. With proprietary software like Microsoft Word, each user may need a license to run their own copy of the application, but Selenium is open source, so does not require the purchase of any license to use.

And as a programming library, any number of users can reference Selenium in their own code and execute it. Users can run multiple automated tests (or other scripts) at once — if they write their program to run in parallel.

But in order to maximize parallel execution (for single or multiple users) you need to have a remote Selenium grid. There is an open source grid that anyone can deploy, but there are also commercial services the host Selenium Grid with additional tools at a cost. These companies include Sauce Labs, BrowserStack, LambdaTest, and Applitools. Each company has their own policy about multiple users, and of the ones I mentioned, they all support multiple users in their own way.

This post is based on a question asked on Quora at:

What can I do to expand my skills beyond testing?

Someone asked about self-improvement after 10 years as a tester and wanting to expand their knowledge into software development.

I can sympathize with this attitude because I went through a similar mindset — which let to my eventual burnout and move to Fiji that I’ve written about previously.

Here is my response:

After 10 years as a tester you probably have pretty good testing skills.

Adding development skills can only increase your value as a tester because it will allow you to communicate better with developers and understand the architecture to find, anticipate, and troubleshoot bugs better.

And if you want to move into development (creative vs destructive role) your skills as a tester will help you and maybe influence other developers to think of testing first.

Other areas you could branch out into and expand your knowledge include operations/devops, understanding system architecture, project / product management, team leadership, or specialized domain knowledge (such as healthcare or machine learning) which can all benefit your work in testing or provide alternate career paths if you’re looking for change.

See the original post on LinkedIn here:

Who cares about this test?

Last week, I wrote about (one way) how to categorize tests.

And I promised a follow up about different ways to categorize tests. Or “how to slice the cake” as I put it.

This is not that follow up.

But after getting some feedback, I thought I’d expand a bit more on the “how” I categorizing tests.

So rather than talk about the different slices of cake, I’m going to talk about “why” I sliced it this way. If you get a smaller slice, it’s not (only) because I think you’re fat. And if you get a bigger pice, it’s not (just) because I like you more than your sister.

The title gives away the “how” — which was pointed out by a reader that I didn’t make very clear in my previous post. That is to say, I categories tests by “who cares about them”.

A developer test is obviously a test that a developer cares about. A user test is a test that a user cares about.

Maybe that’s why we let users do so much of our testing for us.

And so on.

In my current role, there are several gaps in “what” is being tested, and this leads to frustration with QA. The QA team, the QA process, the testers, the test automation framework, etc. It’s my job to help identify and solve this.

My first premise of testing is that you can’t test everything.
Which naturally leads to the second premise:
That there’s always room for improvement.

The question then, is where to improve? And that isn’t always obvious. Or different areas to improve may be obvious to different people. Because those people have different perspectives and different needs.

On a content focused website, it might sense that the content itself — and it’s presentation — are of primary importance. But it might be more or less important that the content can be found. Either by searching (or navigating) the site, or by search engine crawling. One perspective focuses on satisfying existing users of the site, the other perspective focusing on gaining new users. Which depends on your business model or stage.

But there are other concerns, and not just functional concerns. When you talk about non-functional concerns people talk about performance, or security, or usability. But what about operational stability? What about monitoring, error recovery, and risk.

One area I think that is mostly overlooked by QA are operational concerns.

How do the people who deploy and maintain and support your applications benefit from testing?

A quick answer is to have smoke tests.

By “smoke tests” I mean, in the technical electrical engineering sense of the term:

Does the smoke come out when you plug it in? [1]

  • When you deploy a change, how quickly can you detect that you haven’t broken something —
    or done (significant) harm to the system?
  • How can you tell that all parts of the system are working as expected?
  • How can you verify that your change is doing what it was intended to do?
  • Did anything else change and what was it impact?
  • Are there any unanticipated regressions?
  • Are all the parts of the system restored to normal capacity?
  • What (if any) impact did it have on users?

Not all of these points are smoke tests. Where do you draw the line?

It doesn’t have to be a line between “smoke tests” and “full testing”. You can have gradual stages. There may be things that you can report on in:

  • less than 10 seconds
  • under 5 minutes
  • about 1 hour
  • at lease a full day
  • several weeks or more

Again, these types of tests may be doing different things. And different people may care about them.

So I think testers should spend time thinking about the different people who might care about their tests, and plan accordingly. This also means budgeting time to work on tests that mean the most to the people who care about them. Because you can’t test everything.

A good exercise is to find out everyone who cares about the software product and find out what their concerns are. In QA, we often think about two groups of people:

  1. The users (as represented by the product owner)
  2. The developers (as represented by the developer sitting closest to you — or the one who gives you the most feedback).

    Not only should you ask them questions, but you should ask questions of other people including:
  3. Sales – they might care more about the experience of new users,
  4. Compliance & Legal – they might care more about accessibility, data retention, and privacy,
  5. Operations – they might care more about performance under load, security, ability to roll back features, and making sure all parts of the system are communicating.

I’m sure there are lots more people who are about the system.

Conversation on this top sparked one discussion about a real world scenario that probably never would have occurred to a tester focusing on functional requirement from a user perspective who doesn’t understand the architecture — and doesn’t know the deployment process.

The way to address all these concerns in testing is to communicate. And I can’t think of a better way to communicate than to have cross functional teams where all perspectives are represented — and everyone cares about testing, not just the testers, and that testing isn’t something you do just before “throwing it over the wall” to production.

[1] Everyone knows that smoke is the magical substance that makes electronics work. Because when the smoke comes out, it stops working.

How do you categorize tests?

Starting a discussion on test types topics…This will probably become an outline for a presentation.

Not all tests types will be discussed and some are orthogonal, some not relevant (e.g. to QA). Most of all, definitions of test types are fuzzy and vary.

Starting with a simple list…no, starting with my take on categories. (There are 2 types of categories… )
I divide software testing (generally) into:

A. Developer tests and
B. Tester tests

There is a third type of tests also, which I’ll call non-tester tests. Things like A/B testing, usability testing, etc. which are performed with a different goal than verifying functionality or uncovering defects.

C. Other types of tests

There are also tests which might be verifying functionaly or finding defects, but have non-functional requirments, like security testing, performance testing, etc. that need special skills, or are not usually the focus of functional testers.

D. Non-functional requirements testing.

Developer tests include unit tests, integration tests, component tests, etc.There may be different decisions on what makes a “unit test”, whether it can access the file system, network, other classes, etc. (e.g. stict “London School”). But for my purpose I like to group them into tests that are performed by — and primarily “for” developers, and those performed by QA (and other) which are primarily designed with the product or user in mind.

I’m not talking about white-box vs black-box testing, though that category generally overlaps, meaning white-box tests are usually performed by developers. White/Black box has to do with whether you can see inside the system (the code, architecture, etc) or if you treat it as a “black box”, and are not concerned with the implementation. One advantage of white-box testing is that by knowing — or exploring — the architecture, you can better anticipate where to find bugs, but it’s also a case of missing the forest for the trees. Treating the system as a black box can mean focusing what the system “should” do, and not “how” it does it.

While the benefits are similar, and as I wrote the last paragraph, I was tempted to think maybe I do mean white-box and black-box. But not quite. Even if the people performing the tests are the same, and benefits correlate, the methods don’t. That is to say, my category doesn’t depend on whether it’s white- or black-box testing.

When I say developer tests, I mean tests that the developers care about (and nobody else really does), because they only benefit the developers — they help the developers write better code, faster. And so conversely, tester (maybe I should say user) tests primarly benefit the user, or the client. These tests are designed to help know that the product is functioning as desired, and to make sure that there are no (fewer) defects.

But another, more concrete way I break down tests into developer / tester tests (and “tester” doesn’t have to mean someone within a formal QA department) is to say that developer tests are performed on the code itself. And QA tests are perfomed on a delivered, whole product — or System.

In the case of web applications, this means tests that are performed before a deployment, and tests that are performed after a deployment. For desktop applications, that might be the compiled binary — and whatever else it needs to operate. Another way to look at that is to call these type of tests “System” tests.

So we have:

  1. Developer tests and
  2. System tests

However, when you say “System”, sometimes you think of the system architecture, and what you might be categorizing is how the system works — for the reason that we talked about above, vis white-box vs black-box. By understanding how the system works, you’re better able to test the system because you can then interact with it more efficiently:

  • By targeting things that are more likely to break, and more likey to contain bugs. Areas where interfaces exist, complexity is hidden, etc.
  • By interacting with those parts of the system directly (via API, DB query, etc) you can test system states and internal functionality that would be more difficult or slow (if not impossible) to expose from a user-focused test. We want to perform more of these tests because they are more likely to find subtle bugs in the system that are not likely to be exposed or anticipated by users, and because they are usually faster to execute (and less brittle) because they don’t depend on acting only as a user would. It’s also often the case that security exploits are found by accessing the system this way, by bypassing the user interface, or at least using it in a way it was not intended, to get more direct access to the system architecture. SQL injection and buffer overruns are exploits that are examples of using this type of attack vector.

With all that said, I’ve only defined the general categories of tests — by my definition of category. And I haven’t gotten down to definitions and strategy of different types of tests.

There are many ways to slice a cake, and that will be the next topic I tackle.

Selenium Jupiter for tests with less boilerplate code

Following on my recent discovery of Selenium Manager, I checked out another project by Boni Garcia called Selenium-Jupiter.

It uses some advanced features of JUnit 5 (aka Junit-Jupiter) to make tests easier to write with less boilerplate code, part of that is using WebDriverManager to automatically instantiate WebDriver instances without downloading chromedriver, putting it in the path, setting path in properties or as an environment variable, etc.

It starts with an JUnit 5 Extension — similar to JUnit 4 Runners – but you can have more than one extension.

class MyTest { ... }

Then you can pass a WebDriver instance as a parameter to your test method (without having to create the instance — the Extension takes care of that for you.

void testWithChrome(ChromeDriver driver) { ... }

Or you can specify a different driver, for example FireFoxDriver, EdgeDriver, SafariDriver, etc.

void testWithFirefox(FirefoxDriver driver) { ... }

You can also specify whether to conditionally run a test with a specified driver, if docker containers are available, or if a site is up, and many other options available for WebDriverManager.

For instance, run this test only if you’re on a mac

class SafariTest {
    void testWithSafari(SafariDriver driver) { ... }

Or run this test only if your Selenium server is up:

class RemoteWebDriverTest {
     void testRemote(@DriverUrl("http://localhost:4444/") { ... }

Or if Appium is running:

class AppiumTest {
     void testMobile(AppiumDriver driver) { ... }

Are companies getting worse at QA testing?

Melissa Perri posed this question on Twitter:

Aaron Hodder had a great response on Linkedin:

He talks about how companies are giving up on manual testing in favor of automation. Definitely worth the read.

My response about the ramifications of automation vs manual testing (it doesn’t have to be either / or):

There are two mistakes I often see around this:

  1. Attempting to replace manual testing with automation
  2. Attempting to automate manual tests

Both are causes for failure in testing.

People often think they will be saving money by eliminating manual QA tester headcount. But it turns out that effective automation is *more expensive* than manual testing. You have to look for benefits in automation, not cutting costs. Not only is someone experience in developing automation going to cost more than someone doing manual testing, but automated tests take time to develop and even more time to maintain.

That gets to my second point. You can’t just translate manual tests to automation. Automation and manual testing are good at different things. Automated tests that try to mimic manual tests are slower, more brittle, and take more effort. Use automation for what it’s good for — eliminating repetitive, slow manual work, not duplicating it.

Manual testing has an exploratory aspect that can’t be duplicated by automation. Not until AI takes over. (I don’t believe in AI.) And automation doesn’t have to do the same things a manual tester has to do – it can invoke APIs, reset the database, and do all sorts of things an end user can’t.

SeleniumManager (beta) released with Selenium 4.6.0

So I was working with WebDriverManager this morning and one thing led to another, and I ended up browsing the Selenium source repo (as one does) and saw some curious commits (like these):

mark Selenium Manager implementations as beta

fix the framework conditionals for Selenium Manager

Add Selenium Manager support for Linux & Mac

from an old friend Titus Fortner.

I reached out to ask him about SeleniumManager — and it turns out it’s a replacement for WebDriverManager incorporated into the Selenium codebase (written by Boni Garcia, the original author of WebDriverManager, in Rust).

The various language bindings wrap a Rust binary (which for reasons I didn’t ask can’t be cross compiled with Selenium’s frankenstein 3rd or 4th generation custom build tool) so the SeleniumManager binary is packaged with the source.

Very cool, I thought, and then asked when it’s coming out.

Turns out, it was released today with Selenium 4.6.0

Here’s the official Selenium blog announcement:

Introducing Selenium Manager

Selenium 4.6.0 released