Digital Twins and Software Testing
How do we know that the software that we build is correct?
The obvious answer is to test it. But the obviousness stops there, doesn’t it?
Testing software certainly can be difficult. User interface complexities often exacerbate testing because of less-than-ideal separation of user interface concerns from others. Unit tests often are performed at levels too low to be considered end-to-end. With better attention given to APIs today, better separation of UI vs. other concerns can help us to increase test coverage of the end-to-end type, but realizing a better coverage goal is more work, as well, because we now must codify work orchestrations in software, headless tests that invoke business logic in specific business contexts to test business rules in what we hope will be the most common combinations. Exhaustive test coverage requires investment, to be sure.
Let’s pause for just a second, though. Let’s assume the ideal is your reality, and the software that you’ve written is covered to 100% by a set of tests. Every line of code is exercised. Every happy path and every exception.
Is your software well tested?
Not necessarily.
Are your requirements complete? Have you tested them fully, i.e., in all the combinations of transactions in the software that you build, and through every user interface that accesses your platform core? Do you even have a means to analyze change in data to see that 100% of invariant data constraints hold true? Is this means to analyze independent of the code you’ve produced? Have you invested in data sets that give you the ability to test your platform and test the quality of the data it produces and manipulates? How do you know the platform you created worked at the beginning, and that it continues to work now?
I once was part of a team that built a software system to manage a network switch. We had to build the management software system while the physical switch was being engineered in another part of the world, so we needed to implement a switch simulator, a digital twin for the switch being developed, so that we could test management commands that could be issued to the switch. One fellow in the team brilliantly wrote software to reverse-engineer the functional requirements document (on demand, repeatedly, with every new version of the requirements document) and subsequently generate simulator state machine code that was compiled into a software version of the switch so that the rest of us could exercise our code.
When did you last see that type of investment in test of software systems you’ve built?
You might argue that building a simulator of some physical piece of equipment is different than for some transactional system, but I would argue to the contrary. Further, I would argue that having an outside in view of the platforms that we build to look at the health of the data produced in and managed by them is precisely what must be implemented to ensure the software that we build performs correctly in the first place and continues to perform correctly as we evolve what we build.
We’ve often heard that the software that we build is the most succinct statement of requirements that we have — and we’d argue this might be the most appropriate form of documentation of the software that we build.
BUT there is a mind bug here: this assumes that what we’ve built is complete and correct — when what we’ve built might only be a fractional part of a full behavior specification. We need the ability to trace back to some form of software requirements that can be proven correct, and ability to measure the behavior of the software we’ve created via the data it manages in the form of constraints on that data and across the lifetime of our data entities: pre-conditions before some business function is invoked, post-conditions thereafter, and invariants surrounding… with this ability being independent of the code that we’ve built!
Implementing this level of testing is an investment both in data quality and in software engineering that isn’t often made. It may seem a bit of overkill — until the returns of this investment are realized. I’d suggest it is investment in a foundation of data metrics of a business that helps us to know how we work is correct and complete, and that the key performance indicators tracked in our businesses are well grounded.
