Cloudflare outage and testing resiliency
Not to pile on the recent Cloudflare outage, but I want to talk about it from a testing perspective wrt a couple of different aspects. We are seeing an increase in cloud outages lately, affecting all the major infrastructure service providers — Amazon, Microsoft, Google, and now Cloudflare. Cloudflare is somewhat unique in that it is a relatively small company (compared to the others) with an outsized impact on internet infrastructure. There are probably several factors contributing to the recent spike in outages, with our increasing reliance on them being probably the biggest. You just didn’t notice as much previously,…
Read more
Simplifying Test Management Strategies
Here’s a somewhat unstructured rant from a question on Quora:https://www.quora.com/unanswered/What-are-the-test-management-simplified-strategies-for-better-QA-outcomes Structuring tests and maintaining test cases can be a difficult and complex aspect of testing. Here are a few strategies that can be used to simplify test management. First, you can group tests by feature and use tags to organize tests. A well thought out collection of tags can allow to you group them though several cross-cutting aspects. For example — you could have one aspect describing what layer of tests you have — unit, component, integration, system, ui, or api. Another aspect could describe the components in the system…
Read more
BrowserStack Icons of Quality Q&A
I was nominated for the BrowserStack Icons of Quality, and they interviewed me, and to my surprise, they published it (with a few edits): See the highlights on LinkedIn: https://lnkd.in/p/gBgksTnZ Or read the published interview on BrowserStack: https://www.browserstack.com/blog/honoring-iconsofquality-aaron-evans-one-shore/ Here is my full original answers: I’d say the most exciting aspect of my job is the opportunity to work on many different projects, with different technologies. I love to learn. And meeting lots of new people and seeing their different perspectives on (among other things) testing. I think the ability of LLMs — as a better search engine with working code…
Read more
Rubber Duck Productivity
Are you easily distracted?Have a tendency to “go down the rabbit hole” or “get lost in the weeds”?I know I do. I find that when I have someone to work with, even if they’re just watching and occasionally nodding their head when I ask questions that are really directed at my own mental process than anyone else, I work better, faster, and most importantly, stay focused on the task at hand — especially if there is a time limit. I’ve tried pomodoro timers, task lists and focus apps, but none of them seem to work quite as well as just…
Read more
Chaotic Neutral – QA Roles and Alignment
Here are the slides for the talk I gave at the Innovate QA conference in Seattle in August 2024. It was a blast and I met a lot of cool people. The hit of the talk was giving away stickers and polyhedral dice to the people listening to my talk.
Read more
Pair Testing and Other Radical Ideas
This is a link to the talk I gave at QA or the Highway conference in Columbus, Ohio back in June 2024. https://www.youtube.com/watch?v=KdEL2bSHUEU
Read more
Mayonnaise Cake
My daughter came home from a church youth activity the other day with an interesting story to tell. She couldn’t wait to share it with us. They made small chocolate cakes and a delicious butter cream frosting and brought sprinkles, chocolate syrup, and candies to decorate the cakes. And then they ate them. When they were all done there was one plain cake left over, and plenty of decorations, but they were all out of frosting. She went to put the leftover cake in the fridge to take home later. There was nothing in the refrigerator except some old condiments…
Read more
Mainframe testers
I’d love to talk with someone who does mainframe testing. What does mainframe testing entail?How did you get into it?What tools do you use?How much do you work with COBOL, Z/OS, JCL, RPG, PL/I, etc?How do you interface with the mainframe & software — terminals, virtualization, etc. ?What challenges are unique to testing mainframe system and what is common?
Read more
Testing Strategy and Supply Chain Data Pipelines
Reading “Your Data Supply Chains Are Probably a Mess. Here’s How to Fix Them.” A good software testing strategy is like a supply chain data pipeline. The purpose should be to get the relevant data in the right hands so they can make decisions about quality and risk. Here are the common challenges: 1. The actual technical process of developing automation can be overwhelming and you can lose sight of the big picture in the implementation detains. 2. The right data to perform meaningful tests is often locked away in different silos. Whether developer knowledge of APIs or business understanding…
Read more
Recipes and Receipts
Recipes and Receipts Say you’re a wedding planner. An important part of weddings is the wedding cake. And the Smiths want a Dutch chocolate cake for their wedding, with white meringue frosting. So you hire a baker to bake their cake. He asks for the recipe, and you hand him a box of receipts. “What am I supposed to do with this?”, he asks. You explain to him that inside the box are the receipts for last year. After sorting through it you pull out a folder with the word “Jones” written on it. Inside it are all the receipts,…
Read more
Buggy code isn’t the problem
I recently read a post by @TestinGil able the cause of fragile tests:https://www.everydayunittesting.com/2024/08/testing-basic-fragile-tests.htm In this essay, he argues that the problem with fragile tests is complex code. Here is my response: I disagree with both the premise of this post and the solution. While sometimes tests appear flaky due to buggy or complex code, that is not the case the majority of the time. Tests are flaky most often because of limitations in automation — timing issues, etc. Secondarily, tests are flaky due to incorrectly written test code. This accounts for the vast majority of cases. So much so, that…
Read more
Unit tests longer than the code they test?
Here is my response to the following question on Quora: Is it typical for unit tests to have a longer length than the code they are testing?Yep. And that’s a bad thing. There are two things that tend to bloat unit tests. 1. Extensive setup If you find there is a lot of setup code then the problem is that it is difficult for you to test in isolation. Either you are creating complex mocks & stubs — because of too many dependencies; or you are having to handle external state — reading & writing to files or databases, instantiating…
Read more
Choosing testing tools
I’d like to see something that is target / platform / process agnostic. I don’t mean cross browser or mobile app automation tools. I mean something for testing systems — database to cloud, network to user interface. Something that is test framework, reporting tool, continuous delivery platform, project management process, and target environment agnostic. The only thing that comes close is to use low level test runners of the *unit variety, and then roll your own 99%. The library that wraps chrome devtools protocol is such a small part of the decision process, but ends up making so many decisions…
Read more
Testers don’t create value, do they?
When asked what value testers provide you often hear something like the following: I’ll just hire good developers who don’t make mistakes. Developers can test their own software anyways. Here is my response to the question about what value testers provide. Originally posted as a response to this post on LinkedIn by Bridesh about whether testers should be paid less than developers: I like to think of testing as a force multiplier. Maybe testing software isn’t as complex as developing it. But having someone focusing on testing the software allows developers to be more productive as well as produce higher…
Read more
Make sure your tests fail
When you write automated tests, it’s important to make sure that those tests can fail. This can be done by mutating the test so it’s expected conditions are not met, so that the test will fail (not error). When you satisfy those conditions by correcting the inputs to the test, you can have more confidence that your test is actually testing what you think it is — or at least that it is testing something. It’s easy to make a test fail, and then change it to make it pass , but testing your tests can be more complex than…
Read more
Do we really need QA (anymore)?
In recent years there has been a trend towards empowering (or at least encouraging) developers to do thier own testing. Automation frameworks, continuous integration, unit testing, contract based testing, and various other tools and processes make developer testing easier and more effective than ever before. So do we really need testers? Do we need a separate QA depepartment? Obviously, we need to assure quality, but should there be a separate QA role? A good craftsman, after all, should make certain to deliver quality work, and who better to make sure that happens than the expert who designed and built it?…
Read more
DynamoDB might be the right database for your application
DynamoDB was designed to fit Amazon’s pricing model. In other words, the primary architecture goal of DynamoDB is to charge you for data storage and transfer. So it was designed to force you to scan the whole table for every item of data, and to put every attribute in every record. Some clever people have discovered ways to (sort of) make it work for simple scenarios using GSIs and LSIs, but Amazon’s response was to charge you for using them and limit how much you can use them. If you want to find a piece of data in DynamoDB without…
Read more
Test Automation Meetup Talk on Test Frameworks
I gave a talk for the Test Automation Meetup sponsored by Zapple on building and evaluating a test automation framework. Here is a link to the test automation meetup video:https://www.youtube.com/watch?v=7m9NKpJVvQ0 And here are my slides: In this presentation I talk about: What do I mean by “Test Framework” What are the parts of a test automation framework What different tools can be called a framework, or may be considered as part of your test automation framework What are the goals you have for a test automation framework – What do you need it to do, and what do you need…
Read more
QA testing should do these 4 things:
Most of the time, test automation is really only doing #1 — making sure that it works, by navigating around the application, and performing basic functions. This is ok. This is what automation is good at. But you also need to do other things. Things that are harder. Things that are difficult for humans to figure out how to do, and even harder for computers, despite the hype around “AI”. 1. Does it work? As I said, making sure something works is the easy part. Does an application start when I launch it? Can I login? Can I see a…
Read more
What are some Selenium WebDriver locator strategies?
Here is my answer to the question from Quora What are some locator strategies that can be used in Selenium WebDriver? Selenium WebDriver has several locator strategies — or methods for locating elements. Whey you want to find an element, you need to locate it on the page. The way Selenium does this is by using Javascript to parse the HTML source content. In Javascript you can do the following: WebDriver has corresponding locator strategies: It also has additional methods for locating by XPATH, CSS Selector, and link text: XPath and CSS selectors are ways to parse the HTML document…
Read more
How to get started writing unit tests on a large complex codebase without existing tests
A little bit at a time. Developing unit tests for a large scale application can be challenging. Especially if it was written without any tests, or with testing in mind. Writing unit tests requires code to be modular — so you can test a unit in isolation. One of the biggest benefits of writing unit tests during development is not that your code will be well tested, but that it helps you think about writing it in a way that different parts of the application (units) can be tested without interacting with the whole application. But if it hasn’t been…
Read more
What is Selenium? Does it support multiple users?
Selenium is a programming tool used to automate the browser — simulating a real user by opening windows, going to URLs, clicking buttons, filling in forms, etc. Selenium is primarily used for test automation, but can also be used for other things. Selenium is available as a library for most popular programming languages, including Java, JavaScript, TypeScript, C#, Python, Ruby, PHP, etc. It is also referred to as Selenium WebDriver, because there were two different projects (Selenium & WebDriver) which did similar things and eventually merged. Selenium uses the WebDriver protocol (a W3C standard now) to communicate over the network…
Read more
Circumnavigar
You know how a search engine can help you with little things like converting units (like: meters cubed to fluid acres) and get other little answers to things like the weather or internet speed or language translations just by typing in your question. AI prompts like ChatGPT have the potential to really enhance this capability (when it’s not going rogue on you (you know: turning racist, expressing creepy affection, threatening to destroy humanity). Anyway, I’ve thought of another little tool I’d like to have built into my search prompt, or available for a writing tool like Grammarly, or whatever that…
Read more
Bashing React
Bashing React I’ve been a long-time casual detractor of React.js. The occasional snide remark here & there, or long rant in the internet that nobody ever reads. The gist of my complaint is that it’s overly complicated — while at the same time not doing enough to help developers. Other frameworks (like Angular) are complex, but they aim to provide a more complete solution in a coherent form. To be effect (read: develop something beyond a toy app) With React, you need a whole lot of other things with React that you have to piece together. Redux, React Router, etc.…
Read more
What can I do to expand my skills beyond testing?
Someone asked about self-improvement after 10 years as a tester and wanting to expand their knowledge into software development. I can sympathize with this attitude because I went through a similar mindset — which let to my eventual burnout and move to Fiji that I’ve written about previously. Here is my response: After 10 years as a tester you probably have pretty good testing skills. Adding development skills can only increase your value as a tester because it will allow you to communicate better with developers and understand the architecture to find, anticipate, and troubleshoot bugs better. And if you…
Read more
Who cares about this test?
Last week, I wrote about (one way) how to categorize tests. And I promised a follow up about different ways to categorize tests. Or “how to slice the cake” as I put it. This is not that follow up. But after getting some feedback, I thought I’d expand a bit more on the “how” I categorizing tests. So rather than talk about the different slices of cake, I’m going to talk about “why” I sliced it this way. If you get a smaller slice, it’s not (only) because I think you’re fat. And if you get a bigger pice, it’s…
Read more
How long does it take to learn Selenium and Python
This is another post that grew out of a question asked by someone else online. Here is the original question (and my answer) on Quora: https://www.quora.com/How-long-does-it-take-to-learn-selenium-Webdriver-with-Python There are several different things here, and they affect how long it will take you to learn Selenium with Python. Let’s break it down: Your existing knowledge in each of these topics will affect how easy it is. For example, if you’ve already used Selenium with another programming language, that means that (to some degree) you also know programming and test automation principles. So all you need to learn is Python. But if you…
Read more
Skating on Thin Ice
The weather has been warming up in a Montana recently, leading to a slushy mess in our driveway. But my daughter discovered something today when she went out with her mom to feed the animals… We had a hard re-freeze last nice and the driveway is a sheet of ice. Very slippery, so she abandoned her chores and put on her ice skates.
Read more
ChatGPT’s killer feature
ChatGPT has two killer features, and one gimmick (and one real feature) that gets all the press. The gimmick is the ability to compose (synthesize) coherent texts — write a story or essay, which is enabled by the cool feature — it’s language model, which is able to understand and generate language remarkably well ( Is ChatGPT English only?) But the two killer features that most people are discussing are ChatGPT also has one “un-feature” that makes it valuable. It’s not monetized. Google was great, and page-rank was a good algorithm, until people learned how to game the system with…
Read more
How do you categorize tests?
Starting a discussion on test types topics…This will probably become an outline for a presentation. Not all tests types will be discussed and some are orthogonal, some not relevant (e.g. to QA). Most of all, definitions of test types are fuzzy and vary. Starting with a simple list…no, starting with my take on categories. (There are 2 types of categories… )I divide software testing (generally) into: A. Developer tests andB. Tester tests There is a third type of tests also, which I’ll call non-tester tests. Things like A/B testing, usability testing, etc. which are performed with a different goal than…
Read more
Selenium Jupiter for tests with less boilerplate code
Following on my recent discovery of Selenium Manager, I checked out another project by Boni Garcia called Selenium-Jupiter. It uses some advanced features of JUnit 5 (aka Junit-Jupiter) to make tests easier to write with less boilerplate code, part of that is using WebDriverManager to automatically instantiate WebDriver instances without downloading chromedriver, putting it in the path, setting webdriver.chrome.driver path in properties or as an environment variable, etc. It starts with an JUnit 5 Extension — similar to JUnit 4 Runners – but you can have more than one extension. Then you can pass a WebDriver instance as a parameter…
Read more
Are companies getting worse at QA testing?
Melissa Perri posed this question on Twitter: Aaron Hodder had a great response on Linkedin: He talks about how companies are giving up on manual testing in favor of automation. Definitely worth the read. My response about the ramifications of automation vs manual testing (it doesn’t have to be either / or): There are two mistakes I often see around this: Both are causes for failure in testing. People often think they will be saving money by eliminating manual QA tester headcount. But it turns out that effective automation is *more expensive* than manual testing. You have to look for…
Read more
SeleniumManager (beta) released with Selenium 4.6.0
So I was working with WebDriverManager this morning and one thing led to another, and I ended up browsing the Selenium source repo (as one does) and saw some curious commits (like these): mark Selenium Manager implementations as beta fix the framework conditionals for Selenium Manager Add Selenium Manager support for Linux & Mac from an old friend Titus Fortner. I reached out to ask him about SeleniumManager — and it turns out it’s a replacement for WebDriverManager incorporated into the Selenium codebase (written by Boni Garcia, the original author of WebDriverManager, in Rust). The various language bindings wrap a…
Read more
Looking for a Tester with GoLang experience
I was just talking with a recruiter looking for a QA engineer with experience using Go programming language for testing. While Go is gaining popularity – especially among systems application developers (for example, Docker is written in Go) and for developing microservices, not a lot of testers have much experience with Go. That’s because Go is relatively new, and if you’re testing something as a black box (as QA typically does) then it doesn’t matter what programming language you use to write tests in. Go does have testing capabilities — primarily targeting unit testing go ci, and at least one…
Read more
Tests need to fail
Greg Paskal on the “Craft of Testing” Youtube Channel, talks about the trap of “Going for Green” or writing tests with the aim of making sure they pass. He has some great points and I recommend the video. Here are my comments from watching his post: Two big differences I see with writing test automation vs traditional development: 1. Tests will need to be modified frequently — over a long time, not just until it “passes”. 2. Test failures cause friction, so you need to make sure that a failure means something, not just a pass. What these two principles…
Read more
VMWare Cloud Director Security Vulnerability
If you use VMWare vCloudDirector administration tool for managing your virtualization datacenter, you should be aware of the following vulnerability and patch your systems. “An authenticated, high privileged malicious actor with network access to the VMware Cloud Director tenant or provider may be able to exploit a remote code execution vulnerability to gain access to the server,” VMware said in an advisory. CVE-2022-22966 has a CVSS score of 9.1 out of 10. Upgrading to version VMWARE Cloud Director version 10.1.4.1, 10.2.2.3 or 10.3.3 eliminates this vulnerability. The upgrade is hosted for download at kb.vmware.com. If upgrading to a recommended version is not an…
Read more
Something went wrong. Please refresh the page and/or try again.
Fijiaaron 