When we talk about the test product that is owned and managed by the testing team, what comes to your mind immediately. Is it a test suite, test management tool, or any defect tracking tool?

I guess the most obvious one that comes to your mind is the test suite, in this automation era its counterpart can be easily termed as an automation suite, but just adding the new test cases to the existing automation suite and running it once or twice by the testing team at the time of the release doesn’t necessarily make it fall under the category of a product.

If we go by the definition of a product, a product is an object or system made available for consumer use.

In our case, the consumer is the one that is using our automation suite for regression of their application or just to validate that everything is working fine as expected and not breaking something.

Let’s explore the pillars of a great product in the context of an automated test suite.

Product Revision

Key Attributes of Product Revision are Maintainability, Flexibility, and Testability

Maintainability: In the context of the automation suite it means you can modify the existing test cases or add new ones as per the changes in the product with time and the ease of making such modifications in the test suite. How much time does it take? Lesser the time and effort you need to put in, the more maintainable is your test suite.

Flexibility: Flexibility means if your test suite is flexible enough to automate test cases of all the systems under test e.g microservices, app, web.

Testability: This answers the question that do you have any mechanism to test your test suite? Do we have any PR checks, linter checks to test the automated test suite? they are essential to ensure the testability of the product.

Product Transition

Attributes that ensures the Product Transition are Portability, Reusability and Interoperability

Portability: It means that you should be able to run the test suite on any machine whether its a dev machine or any CI setup. It can be achieved by dockerizing your test suite or exporting your test suite as an executable package.

Reusability: Reusability means whether you can reuse the module that you have created earlier e.g reporting modules or common utilities (logging, data fixtures, mocks, logging, etc) to build a new module to test another system under test. Are your existing functions written in a way that it can be used with other modules to automate new functionalities and scenarios?

Interoperability: Interoperability means if your test suite is capable enough to communicate with the other tools for exchanging information e.g any alert and monitoring tool like Grafana.

Product Operation

Key Attributes of Product Operation are Correctness, Usability Efficiency and Reliability, Integrity

Correctness: Correctness means how correctly you have automated your test Scenarios. Do we have the necessary assertion/checks in each automated scenarios to ensure that we have covered it's all critical functionality and a green build gives us the confidence to push that changes in production env.

Usability: Usability means, how easy to run/trigger these tests and to how many systems it has been integrated to, as these factors increase the usability of your test suite. Is your automation suite is integrated with any CI that can automatically trigger these test cases with each changes happening in your system under test, having CI setup or functionality to run the test suite from the developer code increases the chances of the test suite to get used by the devs

Efficiency: Efficiency means how efficient is your test suite to find the bugs, Does the passing of your automated test suite means that all the critical flows of the system under test will work as expected.

Reliability: Reliability means how stable are your automated tests? Does every time the test fails is it due to the actual issue and not the false positive? Any flakiness in the test suite hampers the reliability.

Integrity: Integrity in the automation suite means following the principles honestly whether it’s related to the testing principles while adding assertion or following the standard coding practices.

Until now, all of us are on the same page as to what to expect from a product in context to the automation suite

Evolution of Automation Suite at Grofers

Phase 1: Era of UI Test Suite

UI + BDD (cucumber)

UI + BDD + local device farm

UI + BDD + local device farm +CI

UI+ BDD + local device farm + CI + parallel execution

UI+ BDD + CI + parallel + Dashboard + Running on the cloud

UI+ BDD + CI + parallel + Dashboard + Running on the cloud + optimisation (Debuggability + Speed)