Not to pile on the recent Cloudflare outage, but I want to talk about it from a testing perspective wrt a couple of different aspects.
We are seeing an increase in cloud outages lately, affecting all the major infrastructure service providers — Amazon, Microsoft, Google, and now Cloudflare. Cloudflare is somewhat unique in that it is a relatively small company (compared to the others) with an outsized impact on internet infrastructure.
There are probably several factors contributing to the recent spike in outages, with our increasing reliance on them being probably the biggest. You just didn’t notice as much previously, and the increasingly digitized, interconnected, and subscription-based model of services makes it more noticeable. But also, there are two major factors that I think are underpinning the problem, increased use of Ai and outsourcing of core technical responsibilities.
Both of these are related, but what it boils down to is lack of responsibility, and unwillingness to prepare for (and invest in) contingencies. And that’s where it ties to testing.
You’ve got to give Cloudflare props for their transparency and the amount of detail they are willing to share in their postmortem, which will no doubt lead to improved engineering at the company:
The Cloudflare issue was specifically related to a rewrite and replacement of some of their core infrastructure — memory pre-allocation, which may have been needed to increase the performance & scale of their services, but was not properly tested.
This type of thing is notoriously difficult to test, because it commonly requires infrastructure at scale to test. And there are a lot of moving parts. You can’t find infrastructure scaling issues using Playwright or Selenium to click buttons on a website — which is, unfortunately, where too much of QA testing efforts go.
But this was something that could have been anticipated with boundary testing. And a clear test strategy is described in their postmortem.
The bot management feature configuration file is updated frequenty, processed, and rules are inserted into the database to prevent malicious bots. There had been a fixed number of features (rules) and a bug in the configuraiton processing created an increasing number of rows to be inserted.
So this, couldn’t have been caught in a unit test, but could have easily been tested on small scale infrastructure that did the following:
Generate the feature config
Process it
Check the database
The exact scenario (test data) which caused it to pre-allocate and overwhelm their proxy is not described, but it appears to be a typical memory allocation overrun defect, which is what edge case boundary checking is good at finding.
People have argued about whether Rust (a theoretically memory-safe) language was at fault, or rather, whether the assumption that because Rust checks against buffer overruns means you don’t have to worry about memory allocation — this might be try at the micro-scale, but memory allocation (and leaks) can happen at the macro scale too (see Java).
But the point I want to make is that this highlights
The need for resiliancy, failover, recovery, and monitoring as part of your system architecture.
The need for testing at this higher level. (Or lower level, if you consider infrastructure at the bottom. What I meant was probably “broader” as opposed to “narrower” focused testing.)
Organizations need to be sure that not only does the user-centric functionality of their software work as expected, but that the infrastructure focused aspects of it do as well.
At a dinner meeting with executives at a company several years ago, I was asked why the R&D team (comprising brilliant data scientists) could create prototypes so rapidly, and yet it took so many months for development to produce a working version 1 for customers. Implcit in this question was the assumption that the product engineers were less skilled and whether we should seek to improve the talent of our developers.
I started off by explaining that there were a lot more requirements for scalability, security, and usability than a proof of concept requires when you expose a system to your customers (and the wide-open internet), and that this took a lot of time. I also pointed out that testing takes time, and highlighted some bugs that were oncovered that led to expanded feature scope.
While this company did have an extremely slow and ineffective testing routine (that I had been brought in to help fix), testing wasn’t the primary bottleneck.
The nods of approval from IT and product leadership helped convince the executive team, but I knew that we still had a lot of work to do in QA to remove the friction to help increase development velocity — because we weren’t without blame.
But that conversation did lead to a initiative to spend resources building out a more robust test environment that, thanks to the infrastructure engineers, could be spun up quickly, and that test infrastructure became the basis of their multi-cloud resiliency strategy for production systems.
What are the most exciting aspects of your role as a Test Automation Strategist at OneShore?
I’d say the most exciting aspect of my job is the opportunity to work on many different projects, with different technologies. I love to learn. And meeting lots of new people and seeing their different perspectives on (among other things) testing.
What’s a testing trend/innovation that’s got you excited these days?
I think the ability of LLMs — as a better search engine with working code examples — is really cool. Again, I love learning, and the ability to be more confident delving into a new codebase and exploring things outside of my domain expertise is fun and not nearly as much of a time sink. (I’m easily distracted.)
Also, seeing how other testers are leveraging LLMs to improve their skills, level up with better coding patterns, and tackle things that only senior engineers could do in the past is great. When I see someone write better API tests, or build continuous delivery workflows for test automation who might have been intimidated by such tasks a couple years ago, it makes me excited.
What’s your hot take on AI in testing?
I used to be pretty down on AI. I still think there’s way too much hype. And I can’t stand using agents for coding (like Cursor, Copilot, or Claude code. But part of that is because I’m experienced and opinionated. I might not remember a function name or argument order, so an autocomplete/intellisense/LSP is handy, and all I need most of the time. But when you can’t see your code to edit it yourself it drives me mad. I totally understand why people vibe code (hands off) with an LLM — because you can’t really go half way. It’s either accept everything it slops out, or nothing.
Maybe there’s a future where you turn on the LLM to churn out boilerplate and then turn it off to get real work done.
But for me, the value is really in the power of searching with context and restructuring. At least half of that might be just because advertizing driven SEO has shittified the normal search vector. I remember being amazed by Google and then Stack Overflow in the past too.
But the real hot take now is that I don’t think AI will damage junior coders. I think it’s a huge multiplier for anyone worth their salt — that is people able to research and think for themselves. But the number of people who behave like energy efficient, slower LLMs is too high. That’s my pessimism, my misanthropy showing.
What’s one piece of advice you’d give to someone just starting their career in testing?
Find a mentor, but don’t be a follower. Explore new ideas on your own. Make mistakes, create ugly code, develop bad patterns. But refactor, iterate, learn from that. The hardest thing in technology these days is that there are so many establish, entrenched patterns, frameworks, processes — that are so stupifyingly bad — that it’s hard to see the good intentions in them, or see past the bloat for the value.
When you come up with an idea on your own, and realize it’s trash (maybe by seeing a better version), it’s a golden opportunity to learn and course correct — with wisdom instead of blind obedience. But when you see your unique idea validated, confirmed, or reinvented (before or after you) it’s golden. And gives you the confidence to not just hope for something better, but see the potential of being able to make the improvement.
The best thing you can do nowadays is try to understand the intent of some hopeless broken process or overly complex framework and show others how it can be simplified. You’ll probably be ignored — at least for now, but keep at it.
How do you keep up with all the new trends and tools in software testing?
I just sit back on my porch and yell at all the kids to get off my lawn. And shake my stick at everyone and talk about how hard we had it back in my day, uphill both ways.
Seriously, I investigate trends and tools that interest me, and don’t worry about the rest. With experience comes understanding that you don’t have to be expert in everything, and trying to be can stretch you too thin — or wear you out. Be exceptional at one thing or pretty good at two or three things and be willing to adapt when forced. But don’t always feel like you have to bend to the latest fashion.
For example, if you know Selenium well, you don’t have to learn Playwright, or Cypress, or whatever comes next. There are still plenty of jobs that need Selenium skills. If the job you have declares you have to change, you can learn it if you want — or move on. I know that’s easy to say when you’re old and stubborn, but it applies the other way too. If you’re passionate about a new tool or trend — go ahead and jump in. Cool, now you know two tools, and your perspective is enhanced by knowing the differences and similarities between the old and new.
For me, learning about what’s outside of testing is more exciting these days. I’ve worked with so many different languages, platforms, tools, frameworks, etc. that one more isn’t a big deal. But getting better at devops or platform engineering, or trying to understand what makes sales guys and girls tick, or how scrum masters sleep at night without nightmares of jira, or even experiencing what it’s like to twist your mind in a knot developing a React app. If nothing else, it gives you empathy for people who have different problems in life.
What are the things you wish you knew about software testing when you started your career?
I wish I knew how typecasting it was to accept a job in testing. When I started testing nobody — I mean no one had even heard of software testing — except maybe Cem Kaner. Sometimes I feel like Mark Hammill. For those of you under 50, that’s Luke Skywalker. He was so typecast that he never got another job, except in a horrible D-rate horror movie called “Guyver”. But he learned to become happy as a voice actor. Even poor Alec Guinness (O.B. 1 Kenobi) who was a prestigeous Shakespearean actor and had won an Oscar never got past it — but he rolled with the punches and went to conventions and shook hands with Star Wars fans (and laughed all the way to the bank — he made a fortune getting 2.5% of Star Wars gross profits.)
I’ve been lucky and learned to embrace testing, but only after literally running away to a remote island to get away from it. And now I get to speak at testing conferences and was honored to be asked to give these rambling answers.
Outside of the tech world, what’s a hobby or activity you’re really passionate about?
I used to have hobbies and passions, but I’ve really mellowed out and given up on my dreams. I wanted to be a rockstar or an author or a painter or an extreme sports athlete (surfing, snowboarding, etc) and then I wanted to build a big successful company, but now I just want to water my garden, walk in the woods, and watch my kids grow up and see their dreams crushed too.
Here’s hoping you all can find contentment in the small things in life too.
I’ve been a long-time casual detractor of React.js. The occasional snide remark here & there, or long rant in the internet that nobody ever reads.
The gist of my complaint is that it’s overly complicated — while at the same time not doing enough to help developers.
Other frameworks (like Angular) are complex, but they aim to provide a more complete solution in a coherent form. To be effect (read: develop something beyond a toy app) With React, you need a whole lot of other things with React that you have to piece together. Redux, React Router, etc. While with other frameworks (like Svelte) both radically simplify and improve performance but don’t have the pretension of solving complex developer problems.
Recently, I decided to get professional about bashing React. Not more professional in my behavior, but I wanted to go pro as an anti-React troll.
I decided I needed to learn more about it. To, you know, criticise it more effectively.
So I turned to ChatGPT to start asking clarifying questions. When I diss something, I like to really get my facts straight — usually after flying off the handle and getting them all wrong.
Do I ever publicly apologise or issue a retraction / correction?
HAHAHA.
I used to joke that React feels like it was written by AI.
See, even a couple years ago, when we (normal people) talked about AI, we meant machine learning analyzed uncategorized data and neural network pieced together an algorithm that develops a hueristic for determining if something is, in fact, a picture of a kitten.
(Apparently AI couldn’t figure out how to identify street lights, or crosswalks, or busses — and outsourced the job to us.)
Anyway, the algorithms developed by AI algorithms are often surprising and usually inscrutable. But they worked (for the most part) with a few surprisingly strange exceptions. Which a good reason to use AI to inform, but not make decisions.
What I was saying is that React feels like it was developed organically, without a plan, by a large group of people working independently and it coalesced into something that may be useful for developing web applications, but it is bizarrely confusing to someone looking at it from the outside, and doesn’t have a coherent organizing structure. In other words, it was a cobbled together complex system that came out the unique boiler-room of Facebook’s needs and evolved with additional layers of complexity as it was adopted and adapted by a broader community.
Compared to something like Angular, which has a clear design (even if it was designed by committee and is too rigid and too heavyweight for most projects) that is logical, and it’s largely a matter of taste and (need for complexity and structure) that determines if you want to use it.
I recently watched the React.js documentary:
And was going to say that I feel like I was right, except that React came out of the mind of one person, Jordan Walke. Which is not quite true, because it didn’t take off until a core team of collaborators and contributors and evangelists pitched in. So it wasn’t really a gradual organic design from a large organization, but was a more deliberate design led by a single visionary to replace the organic jumble of tools used by that large organization, Facebook.
In a rare fit of humility, I decided (before watching the documentary) that there must be a reason so many thousands of developers find React so compelling, beyond the fact that it’s promoted by Facebook.
And coincidentally, I was experimenting with building a website out of components using Svelte.js, and feeling quite productive, despite not doing any real development, other than building the (mostly static) site out of composable compents.
The header contains the logo, menu, and login components. The menu takes the topics (About, Blog, Contact, etc) as data properties. The login component has two states (logged in, not logged in) and affects the menu (whether admin menu is displayed). Each page contains the header, main, and footer sections and so forth.
Being able to compose a site from components really helps with structuring your code. But I still didn’t understand all the additional complexity.
Being happy with my progress using Svelte, I thought it would be worthwhile trying React for this simple scenario. And possibly even try to learn a bit about Next.js for server side rendering a static site. The main case I want is to be able to pull content from other sources (CMS, database, or git repo) and render them into a static site, but be able to update page and sections of pages by editing plain text (or markdown) files. In other words — like a blog.
Perhaps someday, someone will invent a useful tool for this.
So I started exploring React as a way to structure HTML. I’ve long suspected that majority of React fans (and users) use it only for this. The claim of React without JSX was disingenuous, because JSX is the primary reason for using React.
People say that React is easy to learn because has a simple interface, but that’s nonsense. React has the most complex interface of any UI framework. Things like componentDidUpdate and componentWillUnmount that only make sense at the most low level for the most complex interactions — and details that almost nobody knew existed in JavaScript at all. I submit that it was actually the complex low level hooks (that React has, but nothing else did) that made it popular for complex sites like Facebook where lots of things are happing below the surface that users aren’t even aware of.
And people talk about the Virtual DOM and how it improves performance. While that may be true for extremely complex DOM interactions (hundreds or thousands of elements changing), but for most sites and applications, it doesn’t. In fact, for most cases, the virtual DOM is slower than just finding and replacing real DOM nodes in place. React’s DOM diffing strategy is actually very basic, and doesn’t (or didn’t) handle things like a <div> tag changing to a <span> tag and everything underneath it stays the same. Or (earlier) adding or removing elements from a list.
While the concept of a virtual DOM was a technical marvel, the implementation left a lot to be desired — and came at a big cost, both congnitively, and code complexity wise, as well and framework size and performance. “Lightweight” frameworks like Preact were create to replace the supposed lighweight React.
Which get’s to the real complexity. React is pretty complex. But it’s benefit comes from not needing to understand the complexity hidden inside — until you need it. Most frameworks can’t compete, not in simplicity, but in being able to adapt to the real complexity when needed.
Reacts “simplicity” comes at the additional cost of React not having a an answer to state management, routing, server side data, and a bunch of other things that are needed for creating a real web “application”. These are all either “extras” or “roll your own”. And you bring in things like Redux, React-Router, Apollo GraphQL, etc to handle these. Which then makes you wonder why you’re using React at all in the first place.
Most people are happy using JSX to compose UIs, and occasionally dip in to properties for passing data, and gradually let the React “expert” handle the complexity of state, lifecycle hooks, and the rest. And that’s probably the Facebook use case. Someone is working on the HTML and CSS for a small widget (like advertisments, or like buttons) and need to 1) make sure it works consistently in all sorts of displays and device. And someone else handles the lifecycle and state interaction.
That means using React, by itself, to compose UIs, is a perfectly acceptable solution.
Except JSX isn’t a good solution. It can’t handle looping, because it can’t handle statements, so you need to use map() it iterate over arrays, and inspecting arrays of object that have arrays can become a nightmare in React, which is actually pretty easy with a straightforward templating solution.
JSX was a nightmare to use for the first few years, because that meant setting up Babel transpilers, Grunt/Gulp tasks, and Webpack, and configuration. Thankfully, create-react-app (created by Dan Abramov) made this simplier, and now tools (like VS Code) can detect JSX and render it properly. Debugging was another major challenge that has improved.
But, I didn’t come here to criticize React, even though I’ve been doing that for a long time. I came here to praise it. Or rather, to explain that it may have it’s uses, and I want to better understand them. And understand why the decisions made to make React work the way it does were made. For that, I need to understand them better myself.
Earlier, I talked about how React felt like it was inscrutably designed by an AI system.
Part of the appeal of React (and many complex modern technologies) is their complexity. It’s that they are hard to learn, and it is a puzzle to figure them out, and you feel like an initiate into a secret society once you finally have obtained your hard-won knowledge. And people want to be like the esoteric experts, so they adopt their fashions.
What I’ve found is that, while I have been unable to understand React by going through tutorials and looking at sample code, part of that is failing to realize that you don’t need to understand it all.
Most uses of React don’t need to use Redux, and most well designed apps won’t use React-Router. And the less shared state that an application has, the better (usually). So a lot of the complexity associated with React applications (that are outside of React itself), are actually unneccesary for the majority of use cases (rendering content and layout on the screen).
What really clicked for me (after spending some time doing due diligence giving React a fair shake) came after reading this post by Dan Abramov (the creator of Redux) a core React team member.
Perhaps ironically (and perhaps not — depending on your opinion of Alanis Morriset or high school English teachers), the thing that really helped me to understand React, was that it was the AI system ChatGPT (which really just sythesizes existing content) that enabled me to understand (and appreciate) React better.
Maybe only an AI can explain a framework allegedly created by AI.
I’m going to be going through my interaction with ChatGPT as I learn, research, experiment, complain about, and use React. Look for more posts and videos. And if you’d like to study React with me, reach out.
ChatGPT has two killer features, and one gimmick (and one real feature) that gets all the press.
The gimmick is the ability to compose (synthesize) coherent texts — write a story or essay, which is enabled by the cool feature — it’s language model, which is able to understand and generate language remarkably well
( Is ChatGPT English only?)
But the two killer features that most people are discussing are
It’s ability to summarize multiple sources (collected through web crawling and data training).
It’s ability to retain context. This is the big one. The ability to ask a question, and then ask follow up questions based on previous questions and refer back to them is the biggest feature, and the one that search engines don’t have.
ChatGPT also has one “un-feature” that makes it valuable. It’s not monetized. Google was great, and page-rank was a good algorithm, until people learned how to game the system with “SEO”. Now Google is worse than Yahoo or Lycos in 1999 with nothing but advertising and SEO spam.
People don’t have the ability (or motivation) to manipulate ChatGPT for monetary gain — yet. And that, with the three other features:
Fijiaaron 