Simplifying Conformance Testing of Services and their Test Doubles

Aka Simplifying Contract Testing

Context

Consider a client (e.g., a messaging mobile app) that interacts with a service (e.g., the corresponding messaging service).

We have three options to test this client.

1. Test the client against a published instance of the service. If the test expects specific responses from the service (e.g., user X has 10 unread notifications), then we need the ability to put the service in a state in which it will provide the expected responses. Also, we need to ensure exercising this ability will not inadvertently affect the published instance of the service. So, this option is seldom or never exercised.
2. Test the client against a custom instance of the service. This option ensures published instances of the service are unaffected and yet allows the client to be tested against the real implementation of the service. Hence, this option is better than the previous option when the complexity of running and configuring a custom instance of the service is simple and easy.
3. Test the client against a test double of the service. This option is based on the common unit testing pattern — replace depends-on-components (DoC) with test doubles. Since test doubles are often independent of the real implementations, they offer most flexibility and ease in terms of setting up the service for testing. Also, since test doubles can be crafted by the client developers, this option allows the client development to happen independent of the service development (as the test double breaks the dependence between the client and the service). While this option is the best option, its effectiveness requires the behavior of a test double to conform to the behavior the real implementation (this is the red arrow in the above figure).

By conformance, we mean the test double exhibits a subset of the behavior of the real implementation.

How do we perform conformance testing?

When client-service interaction is based on requests and responses, we can check a test double conforms to a service by checking, for every request (supported by the test double), the test double’s response is similar (conforms) to the service’s response.

Note 1: A test double may not support all of the requests supported by the service. Hence, conformance testing of a test double should be limited to the requests supported by the test double.

Note 2: While we can check the responses are identical, using the looser notion of similarity is better as responses may have components that are sensitive to execution context (e.g., response id, execution time) and have no bearing on the semantics of the responses (e.g., number of unread notifications).

Note 3: Notes 1 and 2 are few reasons why I prefer the term conformance testing over contract testing.

Step1: Abstract the responses

Since service responses are often structured, treat responses as nested maps and flatten them.

Flattening a map/response

Next, construct different abstractions of a flattened map (service response) by considering each subset of a map as an abstraction of the map.

Abstracting a flattened map/response

In the above abstraction of a response, I have considered info.name attribute/key as a necessary attribute, i.e., the attribute should be present in every abstraction. Hence, info.name attribute/key occurs in every abstraction while other attributes/keys do not occur in every abstraction. Similarly, we can ignore specific attributes/keys while constructing abstractions. Also, we can employ similar constraints to capture data flow/coupling in abstractions. These constraints on abstractions is dictated by how we want to define similarity of responses. In an earlier effort, such abstractions were referred to as structural and temporal patterns [3].

Step 2: Compare responses via their abstractions

To check the test double’s response is similar to the service’s response, consider the difference between the set of abstractions of the test double’s response and the set of abstractions of the service’s response. Every abstraction of the test double’s response that is not an abstraction of the service’s response is a possible extraneous behavior of the test double and suggests non-conformance that may need further examination and fixing.

That’s it :)

Is this an alternative approach to contract testing?

I believe so.

The common approach to contract testing involves recording all interactions (aka request-responses) between a client/consumer and a test double of the service/provider and then replaying against the service/provider. If replay succeeds, then the client and service agree on the contract. If not, a contract needs to be reestablished [1].

Comparing the approaches,

1. In the described approach, while the interactions are recorded, they are not replayed. Hence, the need of replay infrastructure is eliminated. This is a huge win as it simplifies the complexity of both the test infrastructure and the test execution!
2. Both approaches rely on expert developer knowledge to ensure the comparison of responses does not lead to false and noisy non-conformance verdicts.
3. The described approach involves steps to generate abstractions and compare responses using abstractions. While these steps can be fully automated, performing them can involve computing resources. See [2] for details of abstraction technique.

Is conformance testing really this simple?

In terms of the broad sketch, yes, conformance test can indeed be this simple :) In terms of details, it depends on the tests. Here are few twists.

1. Can requests and responses be captured?
2. Can data flow across different requests and responses? Should abstractions capture such data flows? [2] [3]
3. What about abstractions that are irrelevant? [3]
4. At what stage of software development and release cycle should conformance testing be performed?
5. What if test double captures the desired behavior of the service?

Clearly, the above description of the approach cannot be applied as is. Instead, it needs to be adapted to specific contexts. Interestingly, such adaptations can be achieved using mostly existing reusable techniques and technologies. Also, the applications of the approach [3, 4] can serve as templates for such adaptations.

Has this approach been used?

While I have not yet used this approach for services, I have used it in other very similar and non-trivial industrial contexts.

During Window 8 development cycle, my collaborators and I developed this approach and successfully used it to test backward compatibility between USB 2.0 and USB 3.0 bus drivers. Please refer to [3] for the details.

Later on, we also adapted this approach to optimize device compatibility testing, i.e., use only half of the USB driver test suite to achieve 75–80% of test coverage [4].

References

1. Contract Tests chapter in Testing Java Microservices by Alex Soto Bueno, Andy Gumbrecht, and Jason Porter. Manning Publications, 2018.
2. Mining quantified temporal rules: Formalism, algorithms, and evaluation by David Lo, G. Ramalingam, Venkatesh-Prasad Ranganath, and Kapil Vaswani. Science of Computer Programming, 2012. [Shorter Version]
3. Compatibility Testing via Patterns-based Trace Comparison by Venkatesh-Prasad Ranganath, Pradip Vallathol, and Pankaj Gupta. International Conference on Automated Software Engineering (ASE), 2014. (Slide Deck) [Shorter Version]
4. Embrace Dynamic Artifacts chapter by Venkatesh-Prasad Ranganath in Perspectives on Data Science for Software Engineering. Morgan Kaufmann, 2016.