Yesterday, there was a discussion about our testing framework. The discussion ended soon in discussing testing oracles. I create this post to help with that discussion.
What is a test oracle?
Everybody knows about the Sphinx, the oracle in an ancient times. Oedipus went to Thebes, and the Sphinx asked him her riddle. He was able to solve it. But that is not the oracle I am talking about. In the testing world of bits and bytes, we can state the following. A testing oracle is a mechanism by which we determine if the software under test is behaving reasonably.
An oracle is a mechanism for determining whether the program has passed or failed a test.
Cem Kamer
Cem Kamer has another definition, but it is in fact still the same definition. The discussion yesterday was more or less about one type of testing oracle. There are many oracles. I will explain a few of them in this post.
No Oracle
The first oracle that I found is a simple one. Use no oracle. Just run the program you are testing, do not check for correctness but only check if it crashes. You can do this by hand . Just take random keyboard inputs. Call functions via an API and do not check the return values.
This oracle is easy to create and runs fast. Only running this oracle is not enough. This oracle only detects spectacular things like crashes. Be sure, this oracle is not the only oracle you use.
Independent Implementation
Create a separate program that generates the expected results. The test checks if the outcome of the oracle and the system are the same. This oracle is completely customizable for the system under test, so the results can be exact. The disadvantage is that it can be expensive to create.
Consistency oracle
This is an oracle that is also not difficult to create. The oracle compares the results of the system with a “golden master” or a reference. It detects that results change. The oracle can check without knowing what the results should be. Like every oracle, also here are some disadvantages. One of them is that if the system has a legacy bug. It is not discovered, and it can give a false feeling of confidence. This kind of oracle is good in conjunction with unit tests that are verifying an algorithm.
In our case, we have an algorithm in our software that calculates values. The software sends these values to hardware. Unit tests should verify if the algorithm is correct. This means that the integration tests should not verify the correctness of the algorithm. The integration test checks if software sends a value to hardware. The test stores that value to check for it in the next run.
Only when the algorithm changes the test result changes too. This oracle is good for regression testing.
Constraint based oracle
Sometimes a test validates simple valid or invalid combinations. This oracle is the one for such tests. One example is for example that the test can verify if a user’s birth date is before the date he started working.
These are easy checks. Implementing these checks has a low cost. The test code can become invasive. That is one drawback.
Recapitulation
Test oracles are useful while testing. There are many of them. I only did mentioned a few of them. Using a few of them will improve testing. Only using one of them is not a good idea, every oracle has some disadvantages. One oracle to rule them all does not apply here.