This post shares the journey of how I dealt with my own itch to improve testing workflows.

It discusses what problems I faced during works and how I tried to get solutions to them. I started trying existing 3rd-party solutions, then built an open source solution to better fit my own needs. Eventually, its usage in team works motivated me to build a SaaS version.

What is the problem?

Most of my projects are in full stack, which includes both the front end and the backend. The front end usually uses the JS MVC frameworks, such as Angular.js or React.js. The backend is using Node.js stack. The front end connects to the backend using REST API architect.

For web SPA (single page application) apps, they are usually containing many states in a page. To reproduce one of these states, it may need many operations on the page in a certain sequence.

In most cases, the front end is the central interaction point between the user and the backends. The testings of the backend may also need to go through the front end to verify the backend changes are compatible with the front end.

If the depth of the target state is deep, I need to repeat a certain number of manual behaviors in the browser to do verifications. It is time consuming to do this sort of repeated manual tests. It kills my productivity and creativity as a developer.

Testing workflow

Does this sound familiar?

Early try

I love unit tests, and create unit tests to the backend codes, but never for the front end codes.

So I gave it try to use the unit test concept for the front end. At that time, I was using Angular.js as the front end framework for most of the projects. I picked and tried the Angular e2e testing stack, Protractor + mocha. It was the most popular stack to do automated tests on the angular framework.

After some experiments, I realized this style of testing introduces other issues. It requires a tremendous amount of attention to the testing codes. The testing codes are difficult to maintain and doesn’t relieve much the pain from manual testings.

To have the tests to run repeatedly with this framework, it needs to create a lot of mocks for the backend API requests. It assumes the front-end developers to have a very good understanding of the backend, and how it works with the front end. Quite often, there are magic codes added to the tests to pass the tests. These magic codes are problematic and hard to maintain in a long run.

So, with this testing framework, it needs experienced developers to create the tests in a manageable style. Otherwise, the tests would become too fragile to maintain, and contribute more overheads than benefits to the development process. In practice, front-end developers could come from various backgrounds. They can be junior or senior developers with different coding styles.

The other problem is the testing is only verifying some parts of the logic. It doesn’t test using the actual browser behavior. Passing all the tests doesn’t give me enough confidences. I still need to run manual tests to see if it works and looks as expected in the actual browser.

Unit tests passed…

It makes me doubt if it worths writing codes with such kind of frameworks to test the UI codes.

Try again

I am more aligned with what the book Fifty Quick Ideas To Improve Your Tests emphasizes: The tests should focus on exploring the capability for the users.

I started looking for a simpler way to do the front end tests with least overheads to the development process.

There is! The other testing approach is called visual regression testing. It is like applying the WYSIWYG concept in the editor to the testing.

Let the bot spot the embarrassing stuffs quickly

Basically, the idea is to compare the new UI screenshot with the old one. Then it automatically generates a image that highlights the differences between the screenshots.

When there is a difference detected, developers can review the difference image to see if it is as expected. To accept a difference, developers only need to rebase the screenshot without the need to change the testing codes. This dramatically speeds up the testing process. It suits the human nature that our brain is good at reviewing the images than the codes.

It turns out there were already several open source libraries to do this job. I reviewed some of them, picked the one called BackstopJS that seemed to easy to use to do UI testings.

Build a new one for my own itch

BackstopJS gave me a lot of inspirations in how visual regression test can help in the front end testing. I tried to use it to do testings against SPA (single page application) apps. It takes a final screenshot to do visual regression testing at the end of a scenario test. But, it also needs to write freestyle scripts to update the page’s states. This introduces the same maintenance problems discussed in the Early Try section. It is very likely that the freestyle scripts will become messy to maintain in the end, particularly when there are more than one developers touching the testing scripts.

So messed that people dump it eventually

Also I need it to be flexible to take the screenshots and do comparisons whenever needed in any testing steps in a scenario, not only at the final step.

These experiments motivated me to wonder:

Why not formatting the common browser behaviors, such as clicking/typing/select/screenshot etc, into a standard JSON format.

It would be easier to read and update JSON codes than using freestyle testing scripts in a long term.

So I created an open source library called vbot, which since then I have been using for testings for my front end projects.

How my testing process has been improved?

1. No magic codes in the tests.
2. Independent of tech stacks, as long as the app can be used in web browsers, vbot can be used for testing.
3. Use JSON defined in WYSIWYG fashion, much easier to refer to the actual browser behavior.
4. Review and approve the new screenshot, instead of updating testing codes to make the test pass.

baseline/differences/new

automate the tests and watch the playback in action

Add ingredients to team works

Since I am also the user of the tool, it is easy for me to have the first-hand feedbacks. The more I use it, the more ideas I have to improve and evolve it.

When it comes to using vbot in a team, the baseline screenshots need to be shared at somewhere, so that the other team members can run the tests against the baselines.

Git repository was the initial place to store the baseline images. But these image files became too big to store on it for the time being. Furthermore, the screenshots generated for the same UI page from different development environment would be slightly different, due to different development environment setups. Even it is a small difference, it can produce false negative results.

It will be much easier if the testing results can be stored somewhere on a server, and can be reviewed simply via a browser. So it can be more efficient to communicate with the team by sharing the test results.

Imagine a scenario, in which there is a failed test stored on a server, all the browser behaviors have been documented as well as the screenshots. To inform others the issue, it only needs to share an URL link, which the others open to review at the failed test. Even more convenient, they can download the playbook to run the test to quickly reproduce bugs on their local machine when trying to resolve the issue.

Go SaaS

There are multiple benefits in making the vbot as a service on the web. Not only does it provide a place for team members to share their tests, but also it serves as a supplement to the open source version:

• It enables users to create the playbooks via UI, easier for new users to start with.
• Users only need a web browser to run the automated tests against public websites.
• The playbooks, created on the vbot.io, can be downloaded by the vbot CLI tool and run locally with a single command line. Easier to debug issues or reproduce a bug.
• Tests run under the same containerized environment on the server, reducing the odds of getting differences in the same tests due to the variety of development environment setups.
• Playbooks as well as the testing results can be shared with other users, easier for the team to communicate on the issues.

These potential improvements in the testing workflows gave me sufficient motivations to build a SaaS version around it.

vbot.io online test demo

Final words

The journey started in 2016, and the first version of vbot on the GitHub was released in early 2017. As a side project, I spent most of my off hours in doing iterations to make it better and easier to use. It took almost 6 months to build the SaaS version.

I wasn’t expecting I would create an open source library in the testing field, neither create a SaaS product around it. It is an experimental process of building it from the ground up, during which there were many researches and try-errors practices.

I did have some doubts come from inside of myself in the process, wondering “Does it worth?”. The main driven force of this whole journey is that it is a tool built for my own itch. I am glad it goes this far.

You may want to have a try on vbot.io if you need a simple/efficient method to automate the tests for your web apps and improve your development workflow. There are several improvements and features planned to make the testing workflow even easier for developers. Stay tune!

This post shares a bit in the traces of my side project in UI testing. Hopefully, it will be helpful for anyone who want to build their own projects. I will be happy to interact with any thoughts, please feel free to comment.