As described in a recent post, observability of a system is the ability to observe the behavior/state of the system, and this is dictated by what the system exposes about itself. While this seems like an ability of the observer, it is more a measure of the how much of its behavior and state does the observed system expose to the observer.
Given some expectation E about a system, the testability of the system with respect to the expectation E can be defined as the ability to test if the system fulfills expectation E. This depends on
- the relation between the expectation and the behaviors and states of the system and
- the information provided by the system (e.g., thru public interfaces, thru different output channels) about its behaviors and states.
Assuming an expectation can be related to behaviors/states of the system, the ability to test for the expectation boils down to the information exposed by the system about its behaviors and states. In other words, the ability to test if a system fulfills expectation E depends on the ability to observe behaviors and states of the system relevant to expectation E, i.e., testability depends on observability.
If an aspect/part of a system is not testable, then that aspect/part of the system is not observable.
What’s TDD gotta do with all this?
The basic tenet of Test-Driven Development (TDD) is use tests to help build a system that satisfies expectations. We do this by creating tests that check if the system fulfills our expectations and then building a system that passes these tests; hence, we ensure the system indeed fulfills the expectations (at least as captured by the tests).
In TDD, as we create tests first, we determine what information is required from the system to enable testing of an expectation and accordingly design the system’s interface (to expose the required information). In other words, we first determine the information that the system should expose to help us test an expectation and only then implement the system. Hence, we know exactly what aspects/parts of the system can be certainly observed.
As a by-product, TDD identifies the aspects/parts of the system can be certainly observed.
While this is good, it is insufficient because not all aspects/parts of the system that need to be observed may be tested and, hence, they may not be observable. [Also, aspects/parts that need not be tested may be observable; this “excess” may lead to security issues, but we will not discuss them here.]
A simple solution to address this insufficiency is to explicitly test for the observability of relevant aspects/parts of systems.
Test for aspects/parts that need to be observable
The moment we think of the above possibility, we might argue that observability is unlike functional expectations of a system. Specifically, while the functional expectations that drive testability and TDD are often known a priori, what needs to be observed is seldom known a priori. Often, what needs to be observed is determined after the system is deployed and when questions about non-functional aspects need to be answered, e.g., how should we scale the system to take on the increasing load?
I think this argument is false. Just like functional expectations, non-functional expectations of systems are almost always known a priori in all software efforts; at least, in the systems that involve DevOps, e.g., process 1000 requests per second.
Better yet, from the perspective of observability, unlike functional expectations, the structure of non-functional expectations do not change over time. Further, they seldom change across different loosely coupled systems. For example, almost all loosely coupled solutions have non-functional expectations such as process X requests per second. While X might change as the system scales or across systems, the structure of the expectation does not change.
More over, given the generality of the structure of non-functional expectations, the community could catalog the aspects of systems that need to be observed to reason about general expectations. Such a catalog can enable and ease testing for observability of these aspects when developing loosely coupled systems.
If the above observations are indeed true, then it is time to consider observability as a test criteria in the context of loosely coupled (including distributed) systems whose continuous availability and successful operation depends on observability.