Web testing is broken. Selenium should have died 10 years ago. Why is it still a thing?

I figured people haven’t been spending enough time with exponential graphs recently (inspired by this tweet).

Whenever an engineer writes or updates code, there is some chance the changes work exactly as intended in every situation and a much higher chance something is broken. Luckily, there is an expectation that new code is tested before being released to users. Sometimes, testing is fast and easy; if you’re building a button that changes colors when you click on it, you can have a very high degree of confidence as to whether or not it’s working in a minute or less. Over time, the amount a team needs to test scales sort of like this:

Complexity × Update frequency = Testing required

I love that this guy just emits random icons when he’s frustrated.

So just limit those two factors and you’re in the clear right? Well, yes, so long as you’re building something small and predictable. If you just want to throw together a personal website, testing probably won’t be a big problem for you. However, if you’re building a product that needs to grow and evolve to meet the needs of your users, complexity is bound to go up. Historically, companies could compensate by only updating big products like Microsoft Word once a year. They’d still have a lot to test, but if they only had to do it annually it wasn’t such a big deal. That paradigm would be short-lived, however, as just a few years later the internet would change the way software was consumed.

Full disclosure, I was in elementary school at this point and too preoccupied with building the ultimate Magic the Gathering deck and learning obscure baseball statics to notice or care that any of this was happening, so this is all second-hand. On the plus side, I’m starting to think it’s possible I’ve actually become less nerdy over time, if only just.

There is a persistent startup myth that people will steal your idea if you don’t hold your cards close to your chest. In reality, its role is dwarfed by that of your team’s ability to execute effectively and relentlessly.

Anyways, the internet. Companies start to realize they can get a big advantage by updating their software in real time as they learn more about what their customers want. If Company A can release a critical feature its users need within a week of receiving feedback while Company B’s users are stuck waiting a full year, Company A will have a huge advantage, even if the initial version of the product they release is worse. So subscriptions providing access to frequent updates (SAAS) displaced one-time purchases as the standard for delivering software and we all lived happily ever after. Redo itself is open source and will always be free for individuals, but we use the same model for other companies that rely on our app. Only problem is, SAAS apps are:

• Complex
• Updated frequently

If you recall how we determine the amount of testing a team needs to do, we’ve put ourselves firmly in “a lot” territory. As companies scale, it becomes untenable for their engineers to test everything that could possibly break whenever they want to make a change. At this point, they’ll often try to outsource to testers in other parts of the world who are significantly cheaper than the engineers on their core team. This can buy some time, but the fundamental problem remains: the amount that can be tested scales linearly with the number of hours invested, while the amount that needs to be tested scales with respect to the product of complexity and release frequency, both of which will continue to increase. Even if you can cut costs by 80% by outsourcing, if your product continues to grow you’ll soon be paying more than you did to test in-house.

An example of an automated test created using Selenium Webdriver, a browser automation tool for engineers from 2006 that is somehow still alive in 2020🧟‍♂️

Luckily, the same skills engineers use to build their products can help with our scaling problem by “automating” tests. For example, after building a color-changing button, instead of just checking that it works, they’d write new code using a tool like Selenium that allows a computer to simulate clicking the button and validating that the color changes as expected. Once a test is automated, the team can run it every time they want to make changes to make sure they haven’t broken anything. Another problem solved by the magic of code! Or so you’d think, yet none of the four organizations I’ve worked in had automated a meaningful portion of their tests when they hired me.

One problem is the testing ecosystem doesn’t get much love from the open source community that others do. Selenium was built in 2004 and in 2020 it’s still the industry standard for automation. jQuery was released in 2006, peaked in 2013 and has since been surpassed by 17 different UI frameworks. That’s not to say Selenium is the best library out there; I’m using Microsoft’s Playwright, which is vastly superior in almost every way. But in practice, even the best Selenium-style tool won’t save most teams from the UI testing’s biggest pitfalls:

1. Building a maintainable set of automated tests often requires advanced software design skills, but testing is not a sexy problem to work on (🙅🌶️ ).
   If you’re a senior developer, you’d probably rather spend your time on the stuff your team is delivering to its customers. Engineers who do end up working on automated testing are likely to struggle to understand the implications of the design choices they’re making on the amount of work that will be required to keep those tests up to date and
2. Even in the best circumstances, maintaining automated tests is a lot of work (🤖💩⏳💵🗑😩). That’s because SAAS apps are constantly evolving and every time the functionality changes, corresponding tests need to change with it. A well-architected test framework can ensure test upkeep doesn’t exceed that of the app itself (which is very possible if your best engineers are all working on the app while the junior developer builds an automated test framework from scratch), but constant maintenance is inevitable. Unfortunately, tests that break when the product changes are often ignored rather than fixed because…
3. Humans will almost always choose short term payoff over sustainability (🛢⚖️🌎➡️ 🐒🧬🧠💭💡➡️🙈🌊🌪🔥➡️🐵💸). Don’t believe me? Check out this dumbass who founded a startup and spent half of every 24-hour day coding for months to get ahead only to burn himself out, fall behind, and be miserable doing it. I’m good at understanding how decisions about test automation will play out over a period of years not intuitively, but because I’ve seen it happen multiple times and learned to explain those implications ahead of time in order to cover my own ass in the likely event that my team decides to put off automating for “just one more month” to ship that extra feature.

Every. Single. Time.

So, by and large, software testing is kind of a shit show right now. I’d guess for most teams, fixing test strategy is one of the first five items on their to-do list. I’d also guess that two years from now, most of those teams will have exactly the same to-do in exactly the same place unless during that time someone creates a new option that can change the equation. And that’s what we’re here to do.

Redo is a desktop app that anyone can teach to automatically test their website for them. Using Redo, anyone who knows how to use your site can click a button to open a browser, use your site as they would normally, and get an automated test that can run any time to verify its still working. Teams can run manual tests the same way they do today but instead of having to do it all again next time they make a change, Redo does it automatically. You can get tests up and running in minutes, and you only need to update them when the features fundamentally change. Tests and results integrate seamlessly with tools developers already use and understand like Git and Jest with a minimal footprint. Teams will finally be able to focus on delivering for their customers without sacrificing quality. That’s how we will redo QA.

Demo of saving and running a test with Redo ✔️Element and action data is stored as DRY, transparent, JSON and tracked in Git ✔️Test results can integrate with runners like Jest to validate builds.

The question is, are you a Selenium loyalist, or a Redo revolutionary?

Time to choose a side🤴👸⚔️👩‍💻👷‍♂️️:

This graphic is much cooler on our website where it’s animated.