We’ll start out by introducing you to high-level architectural insight. Then, we’ll walk you through the development of the framework.

We’re going to walk through various issues involved in testing the transactions SDK’s and the resolutions to them and relevant examples will be used while walking through the development of the framework. Even though not all technical details of the framework are mentioned in this blog, it definitely attempts to give a holistic picture of the framework.

Couchbase offers transactions in multiple SDKs: Java, Dotnet, and CXX for now and with a plan to support other SDKs in near future. Testing SDKs that offer the same functionality would pose multiple problems during test automation. Test Automation redundancy is the first one that would come to everyone’s mind. Apart from redundancy, we also have to ensure all SDKs have similar implementations of Couchbase transactions. Eg: Error handling is done exactly the same by all SDKs. These are just a  couple of problems. With a major focus on Transactions, this blog will provide various issues we would face while testing multiple SDKs and how we at Couchbase have solved them.

Introduction to Couchbase Transactions

Distributed ACID Transactions ensure that when multiple documents are needed to be modified then only the successful modification of all justifies the modification of any, either all the modifications do occur successfully, or none of them occurs. Couchbase compliance with the ACID properties can be found here.

Transactions in Distributed Environments

Single Node Cluster: Couchbase transactions work on multi-node as well as single-node clusters. However, the cluster configuration should be supported by Couchbase.

Transactions support for N1QL queries: Ensure at least one of the nodes in the cluster has a query service

Couchbase Transactions SDK Testing

During the design phase of the framework, a deep analysis of the test plan and its automation posed us with multiple challenges. Below are a few of the major challenges and their resolutions. Going forward, we will discuss the problem and its solution. We will also see the development progress of the framework along with these problem discussions.

Problem 1: Redundancy Problem

At Couchbase, we currently support transactions in 3 different SDKs: Java, Dotnet, and CXX. In near future, we will be supporting a few more SDKs including Golang. This clearly provides the QE with a redundancy problem i.e we might have to automate the same test case multiple times once for each SDK. 

Resolution: Each test case can be classified into 3 main parts:

1. Test preparation eg: test data, test infrastructure, etc. 
2. Test execution eg: transaction operation execution i.e insert, replace, etc. 
3. Result validation.

A closer look at these 3 parts reveals that the SDK testing is involved only in the test execution phase while test preparation and result validation actually are independent of the SDK i.e it does not really matter which SDK is used. This led us to design a framework that consists of two parts i.e driver and performer. The driver takes care of complete test preparation and result validation. The driver drives the test execution but only abstractly(We will learn more about this below) i.e issues command to the performer and the Performer takes these and performs the actual test execution.

The FIT framework is designed in a client-server model where the Driver acts as a client and the performer as a server.

Driver: Consists of all the tests preparation and result validation. All tests are the classic Junit tests and can be executed either as a single individual test or as a specific test suite or entire tests suite. All the tests are written one and only one time. These tests can be reused for all the SDKs.

Performer: This is a simple application written once for each SDK. Inside a driver, each test is molded in the form of a Java Object and sent to the gRPC Layer. The gRPC protocol does the work of converting this Java Object into a language-specific test Object and sends it to the performer. The performer gets this test Object, reads the instructions, and executes the required transaction operations. Once the transaction is completed, the performer retrieves the result and sends it back to the driver via gRPC protocol

Once the driver receives the result object, the driver proceeds with the result validationTest 

Development Process: Now that we have a top-level idea of how driver and performer operate inside of the FIT framework, let us see the technical aspect of it and how they interact with each other using a few simple example tests.

Eg1: Testing Transactions with a Single Operation: Basic “replace” operation 

Driver code: 

As you can see, all the tests are always written only once and as Junit tests.

Test preparation and Result validation are independent of SDK hence done in the Junit test itself. 

However, the test execution part is done in an abstract way. On the top, it will look like it’s executed in the driver itself. But it engages in distributed computing following the Remote procedure call. The whole test is converted into a Java Object, named as “TransactionBuilder” object in our FIT framework, using the Transaction Builder class and then sent to the performer via gRPC layer using the “sendToPerfomer” method. 

In this example where we are trying to test transaction replace operation, we create a Java object which will have all the details:

1. Document Id on which the transaction is supposed to execute. 
2. Updated value i.e new value which we want the transaction to impose on the doc
3. Transaction operation, in this case, is “replace.”

Once you create such a java object, the sendToPerformer just invokes the gRPC call to send it to the server.

Please refer to Java Performer Code: basicPerformer

So, in the first step, the performer reads the test object and checks for the operation which it needs to execute. In our example, since it’s a replace operation, op.hasReplace() will return true and op.hasInsert(), op.hasRemove() etc will return false.

Inside the replace code block, the performer retrieves the document id, the location of the document, and the updated value for the document. Once all the relevant information is retrieved, the performer executes the transaction i.e ctx.replace() operation.

Once the transaction is successfully executed, the result is sent back to the driver and the driver then similarly retrieves the relevant information from the result object and performs the result validation.

Examples of Functionality tested: This feature of the framework helped us in testing the transactions SDK not just for the doc content but also the transaction metadata i.e expected metadata is present wherever necessary and metadata is removed wherever necessary.

Now that we have some technical insight into the FIT framework, let’s get into a little more detail:

Eg2: Testing Transactions with more than one Operation:

Driver Code:

In this test, the transaction performs insert of the document with docId1 and replace on the document with docId2. So we have to add “insert” and “replace” into the test object and all the required information for each of these operations is sent to the performer.

Please refer to Java Performer Code: performerSupportsTwoOps

Since we have inserted and replaced the op. Insert will return true and the performer retrieves required information and performs insert and then op.replace() will return true and then the performer executes the replace operations and returns the result back to the driver.

Examples of Functionality tested: Initially we did not support all valid multiple transaction operations on the same doc in the same transaction. When added, we could test that functionality with this behavior of the framework. Also, the regular multiple operations of transactions on different documents were tested. Issues like transactions failing to replace/remove documents and expiry were tested well with this support

We have seen in both examples where the transaction is expected to be successful. However, for negative cases scenarios, we expected the transaction to throw errors/exceptions. These errors/exceptions are SDK specific so they need to be handled by the performer. So the driver needs to tell the performer what error/exception to except and the performer needs to do this validation

Problem 2: Error Verification

1. For different causes, the transaction should understand the cause and throw the relevant error/exceptions. So we had to not only test the functionality of the transactions but also the error codes and exceptions are thrown by them.
2. Transaction exception handling is different for each error/exception. eg: Document Not found exception should be handled differently than some transient exceptions.
3. Even for the same exception, the stage of the transaction at which it occurs also results in handling it differently.eg: Write-Write conflict for insert/replace are handled differently than forget operations.

Resolution: The driver should send codes for the causes and exceptions to the performer. The performer will read the codes for the failure causes and induce them using Hooks.

Hooks are internal Couchbase implementations that help to test failure scenarios. In our example below we are just trying to create an expiry before inserting a document

Once failure is induced, the performer will also expect the error/exception this transaction is supposed to throw. So accordingly the performer will retrieve the exception and validate it. If either the exception is not thrown or an incorrect expectation is thrown, the performer fails the tests and sends the failure in the result object to the driver. The driver reads this result object and gives out the expected and the actual failure as output.

Eg3: Test negative cases scenarios:

Driver Code:

So in this test, the driver is telling the performer to execute insert and then to expect the transaction to expire during this insert operation. We send the code “EXPECT_FAIL_EXPIRY” to convey this to the performer.

Please refer to Java Performer Code: performerSupportsErrorHandling

Examples of Functionality tested: All error/exception handling and error codes were tested. Functional testing related to any SDK not supporting or not in sync with agreed functionality of error handling was done. The transaction expiry feature was tested well with this support of the framework.  

Problem 3: Version Management

We have to test different library versions of transactions and the later versions would have new features which are not available in the previous version. So the test framework had to understand which feature is not supported and avoid running those tests. 

Resolution: We have used the Junit5 condition test execution Extensions. Each test suite is annotated with a “@IgnoreWhen” condition. All the conditions mentioned in this will be retrieved and used in the “ExecuteWhen” method we override. Before the driver starts executing any tests, it will contact the performer and get all the functionalities supported by it. The “ExecuteWhen” method will use the information provided in “@IgnoreWhen” and performer capabilities and decide if a test suite needs to be executed or ignored. 

Eg3:

Please refer to Java driver Code: driverSupportsVersionManagement

Examples of Functionality tested: The SDK which developed a feature a bit later than other SDKs, could use this feature of the FIT to turn on these tests once they implemented their feature. This helped us in the test-driven development.

Problem 4: Multiple Performers: Parallel Transactions

Transactions can be executed in parallel. Couchbase transactions confirm the Isolation model.   I.e when two or more transactions are executed on the same set of documents should not lead to dirty writes/reads. To test this, If we randomly run ‘n’ transactions in parallel and in case it causes document corruption, it would be difficult to know what exactly caused the corruption. Each Transaction can have many operations and each operation would have multiple stages. At what operation and what stage did these transactions collide is something we need to know if we need to resolve the issue.

Resolution: We designed a latching mechanism where a transaction executes a few operations or a few stages in an operation and signals the other transaction to start. This first transaction now waits for the second transaction to run and reach the desired stage. Once the second transaction reaches a particular stage, it notifies the first transaction to proceed. This is effectively what happens even for parallel transactions. So we came up with a set of collision points that could lead to write-write conflicts or dirty reads and used the latches to automate these test cases.

Please refer to Java driver Code: driverParallelTransactions

Driver Code:

Examples of Functionality tested/Bugs Found: Concurrent transactions were tested with this support

Problem5: Multiple Performers: Parallel Transactions for Different SDKs

Since we support transactions in multiple SDKs, the same logic can be used while testing the parallel execution of transactions with different SDKs. Eg: Java transactions vs CXX transactions. In the above example, we connected to the same performer since we wanted to run parallel transactions for the same SDK. In this case, TXN A will connect to Performer A(suppose Performer A is using Java Transactions) and Txn B will connect to Performer B (running CXX transactions).

Please refer to Java driver Code: driverMultiplePerformers

Examples of Functionality tested/Bugs Found: Concurrent transactions with different SDK clients were tested with this support. Also helped us in ensuring transaction metadata is intact.

Conclusion

This architectural design of the FIT framework not only helped us in resolving the issues posed to us but also helped us inefficient test automation and helped the transaction development in test-driven development(TDD) mode. 

Efficient test automation: Splitting the framework into a single driver and multiple performers helped us to develop parts of the framework independently. The developer of each SDK provided us with the performer and the QE could focus on the test automation i.e driver. The developers could also add Unit tests into the driver so that all the tests for transactions are handled by this single framework.

Test-Driven Development (TDD): We have developed the java performer and written all the tests needed to sign off the initial few versions of transactions for Java SDK. Once Java SDK was released and the development of other transaction SDK i.e CXX and dot net started, our development team had to develop the performer application while reusing the same driver application. This helped them in developing their SDK in a TDD fashion.

We hope you’ve enjoyed this article. We are adding more features to this framework and will be coming up with a new blog describing the new problems and new solutions. In the meantime, to learn more about the FIT framework, please contact me. To learn more about Couchbase transactions, please visit Couchbase Transactions