Half past nine in the morning, the machine had been running for an hour. We had given ourselves a day to build a small internal service: watch a directory, process incoming files, log every run, raise an alarm when something gets stuck. None of it was new, none of it was hard. By evening it was running, and we had found ten bugs.
The test suite had found none of them. It was green all day, including the hours in which the service was demonstrably doing the wrong thing.
This is not a case against tests. The suite did exactly what we had written it to do, and it stopped us from breaking things while we were moving parts around. It just could not see these ten. Every single one came from letting the thing run - from someone starting the service, handing it a real file, and then going to look at what actually happened.
Three of them are worth telling, less for the bugs themselves than for what they share.
A permission that never shows up locally
The service wrote its results into a directory it had not created. On your own machine that is invisible: everything runs as you, the folder is yours, and the question of whether writing is allowed never comes up. In the test environment the service ran as its own user, and the directory came out of a setup script owned by somebody else.
The test had never asked that question, for a very understandable reason: it created its own directory, in a temp path, right before calling the service. A directory the test process just made belongs to the test process. It could not be anything other than writable.
The lesson is not that permissions are fiddly. It is that a test which builds its own environment always builds a convenient one. That is precisely what makes it fast and repeatable - and precisely why it cannot see this class of bug.
A checker that uses the same directory as the writer
The second one was the most uncomfortable, because it looked like confirmation. After every run, a small routine checked whether the output file was where it belonged. It reliably reported success.
It reported success because it resolved the path through the same helper the writing side used. Both asked the same place where the file should go, and both got the same answer. When that answer went wrong - a config did not take, the path landed one level too high - the writer wrote in the wrong spot, and the checker looked in exactly that spot. It confirmed what the writer had done. Both were off in the same direction.
A checker that inherits its assumptions from the thing it checks is not checking. It is repeating. In miniature, that is the same question that matters in review: whoever looks has to arrive from a different direction than whoever built, or they see the same things and miss the same things.
A log that is not writable while the alarm looks healthy
The third one was the quietest. The log went to a file left over from an earlier version of the setup, one the service no longer owned. The logger swallowed the error, as loggers do - a service should not die because it cannot get a line out - and wrote into nothing.
The alarm stayed green. It was built to report the state of the process: running, responding, nothing unusual in its resource use. All of that was true. Nobody had ever asked it whether a line had been written in the last hour. Two silent days would have looked, from outside, exactly like two healthy ones.
An alarm that reports a process is alive does not report that it is working. That reads as obvious once it is written down. It still goes unnoticed for as long as the process really is working.
The pattern
The interesting part is not the ten. Ten bugs in a day is an unremarkable number for a freshly built system, in either direction. What is interesting is where they all sat.
Not one of the ten was inside a function. They sat in the seam between the program and everything around it: the file system, the users, the permissions, the clock, the process next door. A test replaces that surrounding world with a manageable version of it. That is not a flaw, it is the point of the exercise - it is why the suite runs in seconds and why it tells you reliably, on every refactor, whether the logic still holds. It is also why the suite is blind at that seam.
The conclusion is not to write more tests. It is not to build integration tests for everything either, because those are expensive and end up assembling their own manageable world.
What we took away
Three things, all unspectacular.
Let it run early, somewhere that is not your machine. The half day in which something exists only locally is the day when every assumption about the environment quietly moves in.
Have someone other than the builder go and look. Not as a ritual, but because the author inevitably looks where they believe they put the thing.
And ask for the evidence, not the status. “Is it running?” gets you an answer based on an assumption. “Show me the last line in the log” gets you an answer based on something that actually happened. The difference between those two questions accounted for roughly half the day’s haul.
The next morning the test suite was still green, and rightly so. The ten bugs were never in what it covered. They were in the things we thought too obvious to write down.
