Visual UI Regression Testing with WebAssembly and the Kantu Selenium IDE

The new versions of Kantu for Chrome and Kantu for Firefox bring a major update - they can now run automated visual UI tests inside the web browser , powered by WebAssembly. To our knowledge, this makes Kantu one of the first ( or even the first? ) browser extension that uses WebAssembly. But technology aside, this video shows how it works:

The visual UI testing commands VisualAssert, VisualVerify and VisualSearch allow you to write automated visual tests with Kantu - so Kantu has its own pair of " eyes" now. A huge benefit of doing visual tests is that you are not just checking one element or two elements at a time, you’re checking a whole section or page in one visual assertion. The image comparison algorithms that we use are error-tolerant, and can ignore resolution, size and position differences.

The visual test commands take a screenshot of the page and search it against the provided image. If the algorithms find the image inside the screenshot, then visualAssert succeeds, otherwise it fails. You can supply a confidence value behind the @ symbol. Its value ranges from 1 (exact match) to 0.1 (almost everything matches). The visual test commands wait!timeout_wait seconds for the image to appear. If the image is not found, an error is triggered (you can overwrite this with!errorignore).

The difference between visualAssert, visualVerify and visualSearch is what happens if no image if found:

• visualAssert - Macro stops with error if no matching image if found.
• visualVerify - Macro logs warning if no image is found, but macro execution continues.
• visualSearch - The number of matches is filled into a variable, just like thesourceSearch command does. If no image if found, the number of matches is 0.

Tip 1: UI testing is especially helpful with responsive content at different viewport sizes to ensure changes made at one screen size won’t break a layout on another screen size. You can use theRESIZE command to resize the browser window to emulate various screen resolutions.

Tip 2: Because the visual checks wait for the image to appear, they are also useful for general web automation, and can visually make sure that a web page - or a certain part of it - is loaded and visible before proceeding. On complex websites, such a visual, human-like "page is loaded" check is often more reliable - and certainly easier to understand and implement - then the classic method to wait for the browser's page load event(s) to fire.

The image search runs 100% local in your web browser. It uses C++ powered computer-vision image processing, running directly in your Chrome and Firefox as WebAssembly code. The input images for the visual tests are stored in the " Visual" tab:

When you select a visual test command, the normal Selenium IDE "Select" and "Find" buttons turn into " Select" and " Find". " Select" allows you to take new screenshots for use with visionFind, and " Find" starts an image search. The found areas are marked along with their confidence value. The best match is marked with a pink frame.

At the top of the " Visual" tab you find buttons to take screenshots, and export and import all images. And next to every image is the option to rename it, to add its name to the visual command in the table above (+ symbol), to view it in original size ( symbol) and to delete it.

visionLimitSearchArea (viewport/full/element:locator)

The visionLimitSearchArea allows you to change the search area from viewport (the part of the website that is currently visible in the browser) to full page. And with element:locator you can restrict the image search/comparison to a specific DOM element. This is especially useful for testing canvas elements like Google Maps, Youtube videos or e. g. seat reservation applets.

The screenshot that is used as input for the image search commands is available as "_lastscreenshot" in the screenshot tab. This can be useful for debugging, so you can see on what image the image search was done.

Click on Elements with #elementFromPoint(${!imageX}, ${!imageY})

The coordinates of the center of the best found match are available in the internal variables !imageX and !imageY . Together with the elementFromPoint (x,y) function of the web browser, you can find out which element (locator) is at the image, and then use this locator as input for all commands that take a locator as input. For the most common use of elementFromPoint with the !imageX/Y we introduced a shortcut notation of just using #efp , which of course e-f-p stands for e_lement_f_rom_p_oint. This short video shows how to use #efp to click on an instagram like button:

Testing Canvas Elements

For UI testing and automating canvas elements you need to useClickAt with elementFromPoint instead of Click. Only ClickAt can click on specific coordinates inside a canvas element. By contrast, Click #efp always clicks in the middle of the DOM element, and with a canvas element that means, it would always just click in the middle of it.

Command Target Comment visualVerify (image name here) This sets !imageX and !imageY if image is found ClickAt #elementFromPoint ( ${!imageX} , ${!imageY} ) comment: same as using #efp, see below: ClickAt #efp shorthand notation Type #efp elementFromPoint can be used with any command that takes a locator as input ClickAt #elementFromPoint ( ${anyOtherVariable} , ${var2} ) Here we need to full command as we use different variables as input.

Note that the elementFromPoint function of the browser is not always reliable, on complex websites with overlapping elements it might return an element that can not receive a click. If you find such cases, report them to us in theforum, there are probably workarounds available.

Visual UI Test Example

Quite literally, seeing is believing. The best way to understand visionFind is to run the included DemoComputerVision macro. It is installed with every fresh Kantu install and shows a visual test of the best online OCR website.

Command Target Pattern/Text open https://a9t9.com/ visualSearch buttonImage matches

Works in

Kantu for ChromeSelenium IDE, Firefox IDE Classic

Related Demo Macros

DemoCanvas, DemoComputerVision, DemoVisualUITest