I’d love to talk with someone who does mainframe testing.

What does mainframe testing entail?
How did you get into it?
What tools do you use?
How much do you work with COBOL, Z/OS, JCL, RPG, PL/I, etc?
How do you interface with the mainframe & software — terminals, virtualization, etc. ?
What challenges are unique to testing mainframe system and what is common?

Reading “Your Data Supply Chains Are Probably a Mess. Here’s How to Fix Them.”

A good software testing strategy is like a supply chain data pipeline.

The purpose should be to get the relevant data in the right hands so they can make decisions about quality and risk.

Here are the common challenges:

1. The actual technical process of developing automation can be overwhelming and you can lose sight of the big picture in the implementation detains.

2. The right data to perform meaningful tests is often locked away in different silos. Whether developer knowledge of APIs or business understanding of requirements.

3. A common data communication language is necessary to communicate business priorities to QA and for QA to communicate their findings in a way that can provide contextual meaning to various stakeholders.

4. Different parts of the organization have different goals, and aligning them all with meeting customers need for quality and business need for the bottom line.

Solutions:

1. Leadership should drive the need for QA by communicating thier priorities and assigning test resources according to business goals and value.

2. QA should be demand driven. Developers, product owners, and leadership should seek the knowledge they need to make decisions from testers and enable them with the information needed to accomplish the testing.

3. Testers and developers should understand the business domain language and common communication channels should be open. If test reports and continuous delivery jobs are ignored, find out why: is the data accurate, relevant, timely, and meaningful?

4. QA leadership should align testing with customer and business leads and approach testing as a source of information for product decision makers not as a gateway or “check” on software quality.

The data you need to perform tests and the data you need to make decisions about quality are inter-related but not identical.

The ability to share (and transform) data for testing is critical, but I don’t think a unified tool or process is the solution. It’s why complex ERP deployments fail and why everyone hates Jira.

Recipes and Receipts

Say you’re a wedding planner. An important part of weddings is the wedding cake. And the Smiths want a Dutch chocolate cake for their wedding, with white meringue frosting.

So you hire a baker to bake their cake. He asks for the recipe, and you hand him a box of receipts.

“What am I supposed to do with this?”, he asks.

You explain to him that inside the box are the receipts for last year. After sorting through it you pull out a folder with the word “Jones” written on it. Inside it are all the receipts, invoices, and billing hours from the Jones’ 50th anniversary where they renewed their vows. A very touching ceremony, you remember.

“My accountant makes me keep everything for taxes,” you explain.

After rifling through the folder and, you pull out a receipt and hand it to the baker.

“Here you go, this is for a cake we made last year.” On the receipt are all the ingredients for the Jones’ cake:

10 lbs. flour
1 dozen eggs
1 bottle red food coloring
200 paper plates

When you see the confused look on the baker’s face, you elaborate: “It was a red velvet cake. Red velvet is really just chocolate with food coloring, you know. Don’t add the food coloring.”

“Or the paper plates and plastic forks?”

“Naturally,” you reply. “This is a fancy wedding. Don’t worry about the tableware, just bake the cake.”

But the baker has further objections. “How am I supposed to know how much of each ingredient?”

“Oh, there’s an invoice in the folder detailing the correspondence between the Smiths and the previous baker. You’ll find the number of servings and can just calculate the difference. The Jones had a big sheet cake, I remember, so they really could have just make smaller slices.”

The baker makes a face, but decides he can estimate the proportions for the Smith’s 3 layer cake.

“And the merengue?” he asked.

“The Jones had butter cream frosting. Personally, I like that better.”

The baker gets to work on the, only realizing after he purchased the ingredients that Dutch chocolate is totally different from normal cocoa powder.

He does a great job, and the cake looked great and tasted delicious. You don’t know why the bride made that face when she took the first bite, but Mr. Smith didn’t seem to care and was happy with it, and even happier when he saw the bill came in under estimate.

And business is booming. One of the guests at the Smith wedding loved the cake so much that she hired you to plan her daughter’s wedding. She wants a 7 layer White Chocolate sponge cake with raspberry filling.

You call up the baker with the good news.

“Do you have a recipe for this cake?” he asks.

“What happened to the recipe for the last cake,” you ask. “Didn’t you write it down?”

“Hello?”

I guess I’ll have to find another new baker, you think. No worry though, I’ve got all the receipts for the Smith’s cake too. I’m building up quite a recipe collection.

---

The analogy here I’m trying to make here is between test cases and user stories. Tests are like recipes. User stories are like receipts — or at best, todo lists.

Tests should represent the system, as it is now. And stories represent the tasks that needed to be done to get the system to the state it was in — at some point in the past.

The most recent user stories should represent the system as it exists right now, but they don’t describe the whole system. They describe a change set of the system from the point before to the point it is now. And now will not be the point it is at later.

You cannot build functional requirements from user stories or tasks.

A story can reference the tests needed to verify that the system is working after the story is complete — and that’s a good thing. But tests cannot reference a story without the risk of the test becoming outdated or incomplete. You need something different to describe the system as a whole as it is in the current state, or at any particular state in the past.

If you “add an abstraction” — requirements — you now have tests that verify the requirements, and stories that describe the work done to fulfill the requirements.

However, I’m not advocating for complex requirements documentation. That’s a large part of the resistance to specifying requirements. But another large part is that people feel like they are doing that when they document the requirements in a task management system like Jira.

Can’t tests be the requirements? That avoids duplication. Double plus if the tests are also executable specifications. Automated tests and requirements in one go.

Technically that’s possible, but technically speaking, that’s very difficult. It’s actually more difficult to make executable specifications that to link automated test results to a static specification. And it only works if the specification used to execute tests is the same specification designed by the product team. And practically, that never happens.

I recently read a post by @TestinGil able the cause of fragile tests:
https://www.everydayunittesting.com/2024/08/testing-basic-fragile-tests.htm

In this essay, he argues that the problem with fragile tests is complex code.

Here is my response:

I disagree with both the premise of this post and the solution.

While sometimes tests appear flaky due to buggy or complex code, that is not the case the majority of the time.

Tests are flaky most often because of limitations in automation — timing issues, etc. Secondarily, tests are flaky due to incorrectly written test code. This accounts for the vast majority of cases. So much so, that it is usually justified to “blame the test” and understandable why all other causes get lost in the noise.

The third most common source of test flakiness are environment related issues — either the test infrastructure environment or the system under test. This is an addressable problem but since it is often not under control of either the testers or developers, it is often neglected. Typically this is an operations issue, but devops are cranky, and we should excuse them for being dismissive from having encountered problems of the first and second type so often.

Finally, system complexity (not code complexity) is the real issue with creating test complexity. Having to automate a complex workflow — that need not be so complex — is a real problem, and exacerbates the problems of inherently brittle automation and poor quality test automation code.

One way to alleviate this is to simplify tests, for example, by testing functionality at the API layer instead of the UI layer where possible. Or by using lower layers (API or DB) for data setup and validation.

But my main complaint here is with putting the cart before the horse. If you have flaky tests that fail because of defective code, the problem isn’t that the developers have written defective code and the solution isn’t that they need to test it better — that what testing is for!

If your tests are finding bugs in code, that is their intended purpose. A test that does not find a defect is wasted effort. We do not always know which tests will find bugs, so this waste is expected.

The theory is that QA provides value by being able to look at the result of developer code and more efficiently find defects than the developers themselves. If that is not the case, then the problem is QA, but I don’t believe that. I believe that dedicated testers provide a fresh perspective, and have specific goals and incentive to find defects in a way that developers cannot — and that it can be done cheaper.

That doesn’t mean that developer tests are not also valuable. They are, but should not be expected to catch everything.

The issue is, I think, that tests that are themselves flaky, slow, or uninformative make the effort of finding real defects too costly, and is something that should be addressed.

Simplifying systems reduces the opportunity for defects, obviously, but is really outside the scope of the problem.

Here is my response to the following question on Quora:

Is it typical for unit tests to have a longer length than the code they are testing?

Yep. And that’s a bad thing.

There are two things that tend to bloat unit tests.

1. Extensive setup

If you find there is a lot of setup code then the problem is that it is difficult for you to test in isolation. Either you are creating complex mocks & stubs — because of too many dependencies; or you are having to handle external state — reading & writing to files or databases, instantiating other objects, configuring environments, etc.

In which case it’s not really a unit test. Which is ok, those tests may have value, but you should probably put them in a different bucket and run them at different times.

2. Overly complex tests

If there are many steps to perform (that are not strictly setup) to accomplish the goal you need to test, then there may be an abstraction you’re missing. Or your code is tightly coupled.

If there are many validations needed — then you’re probably trying to test more than one thing, and need to think about what the goal of this particular test is. You don’t necessarily need to test that every value in your result is correct, or inspect that every expected method has been called in a single test. It can be multiple targeted tests.

If you find that you are duplicating a lot of steps or that you are wanting to validate a lot of things in your tests, it may be because setup is complex, or execution is slow. Again, you may not be dealing with unit tests.

The main goal of units tests is not that your code is tested. Test coverage metrics are garbage anyway. The goal of writing unit tests should be that your code is testable in small units. And that may mean refactoring and thinking about it in terms of independent components. Which is a good thing.

Having good unit tests enables safer refactoring. But refactoring is often needed to have good unit tests. It’s not a catch-22, it’s an iterative process. You may start with a bloated end-to-end test. But being able to run that confidently allows you to make small internal changes until you can have small, isolated units that are quickly and easily testable.

I’d like to see something that is target / platform / process agnostic.

I don’t mean cross browser or mobile app automation tools.

I mean something for testing systems — database to cloud, network to user interface.

Something that is test framework, reporting tool, continuous delivery platform, project management process, and target environment agnostic.

The only thing that comes close is to use low level test runners of the *unit variety, and then roll your own 99%.

The library that wraps chrome devtools protocol is such a small part of the decision process, but ends up making so many decisions for you.

The trick is to not let it.

In recent years there has been a trend towards empowering (or at least encouraging) developers to do thier own testing. Automation frameworks, continuous integration, unit testing, contract based testing, and various other tools and processes make developer testing easier and more effective than ever before.

So do we really need testers? Do we need a separate QA depepartment?

Obviously, we need to assure quality, but should there be a separate QA role?

A good craftsman, after all, should make certain to deliver quality work, and who better to make sure that happens than the expert who designed and built it? And software development is craftsmanship.

There are several reasons why I think that QA should still exist as a separate role, but it boils down to 2 major arguments:

1. Perspective
2. Time

Perspective

The saying goes that “you can’t see the forest for the trees” and this applies to developers and testing — they are too close to the product to spot it’s flaws. Or looking at it another way — they are focused on creating something, not on finding out how it can be broken. Much like a writer needs an editor, and an artist needs a critic. Having someone with a different perspective is valuable for seeing things someone intimately involved in the creation process may not see. There is definitely value in having an expert examine the system, but there is also value on having someone inspect it with a naive approach.

And then there is the issue with my statement above about a craftsman judging his own work. Even if my premise about a naive perspective bringing value in testing is wrong, the problem lies with the assumption that a single craftsman creates his productx in isolation. Many software projects are so big, that no one person can understand the whole scope of the system, so one person can be responsible for ensuring that everything works together correctly.

And it is primarily in the cracks between different systems (or parts of a system) that bugs are found.

Time

Secondly, developers don’t always have the time to fully test their software, and even if they make the time (and ignoring any issues about perspective), they question becomes — is this the most effective use of their time?

Developers are experts, not only in the craft of making software, but also in the systems they are making software for. As such, their time is valuable – – and why they are (and should be) handsomely paid for their expertise.

Having someone whose expertise is different (if not less) perform the more labor intensive, repetitive tasks where less expertise (at least in software development) is required only makes sense. The CEO isn’t expected to do everything at a company, even in situations when they might actually be able to. Doctor have nurses, technicians, and other assistants to help them maximize the value of their expertise, and thus the time they can spend on their specialty, and developers should be no different.

My Perspective on QA

I look at my role in QA as providing a fresh perspective, but also especially providing a way to maximize the productivity of developers, product owners, and others by providing the services that they might very well be able to do on their own, but would slow down their velocity if they had to.

I look at test automation as an extension of that. The goal of test automation is to make testers more effective, and relieve them of the boring drudgery of repetitive regressions, that might otherwise cause them to miss important defects, and that allows them to do creative, exploratory, destructive testing.